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This report discusses the results of the College Assessment 
of Academic Proficiency (CAAP) at the College of DuPage (Illinois) . The CAAP 
is a national exam that assesses general learning in six areas: critical 
thinking, science reasoning, writing skills, essay writing, reading, and 
mathematics. Based on data from a total of 3,322 students tested between 1999 
and 2001, results include: (1) entering first-year students at DuPage are 

significantly below four-year public college freshmen in reading skills, 
mathematics skills, science reasoning, and writing skills; (2) DuPage 
sophomores' mathematics averages significantly exceed the two-year public 
college norms; (3) completing sophomores are similar to other two-year public 
college sophomores in their writing skills, but still significantly lower 
than their four-year public college sophomore colleagues; (4) DuPage’s 
students are functioning at levels similar to two-year freshmen and 
sophomores, but below the four-year public college level in essay-writing 
skills; (5) the critical -thinking skills of DuPage students, while similar to 
both entering first-year students and completing sophomores to their two-year 
public college colleagues, do not change significantly over their studies; 
and (6) DuPage students perform significantly lower in their science 
reasoning skills than four-year public college students, but, like most of 
the other skills, their abilities in science reasoning improve during their 
tenure at DuPage. (Contains 40 tables.) (EMH) 
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Chapter 

\ Assessment of General Education 

Background and history 



Institutional level assessment of general education at College of DuPage (COD), or any 
community college, is a challenging task. Consideration of the challenges can be divided into at 
least two distinct sets of issues. One set of issues focuses on defining general education 
outcomes and then operationailzing that definition with appropriate instrumentation. A seoond 
set of issues focuses on implementing the measures defined in the first focus. Responding to 
these challenges, Student Outcome Assessment Committee memberships have implemented 
three annual cydes of representative testing of general education skill. This report provides 
feedback to stake holders concerning our findings. 

This marks the third report in this series. Data from six rounds of testing using ACT's Collegiate 
Academic Assessment Profile (CAAP) are used. These data are drawn from a random, stratified 
sampling of students in credit courses at the 100 or higher level. This sampling is a very strong 
set of data representative of the college's student body. Statistical controls are utilized to identify 
and control for characteristics that are known to impact the validity of comparisons among 
groups induding student "stop-out" and reverse transfers. The following analysis indudes 
examination and support for combining the three years cohorts. This aggregating provides 
suffident numbers of cases to support parametric statistical analysis. As a part of the analysis 
differences among the three cydes are examined and reported on where significant differences 
are found. 

The following report indudes repetition of many explanations from previous reports. Thus, a 
current understanding of the assessment findings from the testing project does not require 
referring to previous materials. 

Challenges to assessment at College of DuPage 

As with any inquiry, the first steps involve identifying what one intends to study. Within the 
general assessment framework, student learning needs to be examined through multiple 
measures at multiple levels from dassroom through institutional levels. Activities at dassroom 
and disciplinary levels have been initiated and are described in other documents of the Student 
Outcome Assessment Committee. When considering general education assessment, two limits 
need to be addressed: what are the knowledge, skills, attitudes, values, and characteristics being 
assessed and who is eligible within the population as representative of the cohort being 
assessed? 

Answering the first question, "What is general education?" is a challenge faced at any 
institution attempting general education assessment. Discussion and debate as to a meaning for 
"general education" can occupy years and careers. A choioe made by members of the Student 
Outcome Assessment Committee was to operationalize some inquiry, aooepting that future 
discussions and alternative measures might result in other assessment formats. Thus, these 
rounds of assessment of general education at College of DuPage utilized the following premises. 

> General Education is learning which occurs as a result of multiple experiences and 
courses throughout a student's educational activities, not a specific learning resulting 
from a sequence of identified courses. 
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> General Education is defined as those ideals expressed in the following statement 
from the college catalog. 



The aims of general education are to enable students to understand and appreciate 
their culture and environment; to develop a system of personal values based on 
accepted ethics that lead to civic and social responsibility; and to attain the shills in 
analysis, communication, quantification, and synthesis necessary for further 
growth as a lifespan -I earner and productive member of society. 



Consideration of this published statement results in identification of seven competencies that may 
be taken as an operational definition of general education at College of DuPage. 

1) An aim of general education is to enable students to understand and 
appreciate their culture 

2) An aim of general education is to enable students to understand and 
appreciate their environment 

3) An aim of general education is to develop a system of personal values 
based on accepted ethics that lead to civic and social responsibility 

4) An aim of general education is to attain the skills in analysis necessary for 
further growth as a lifespan-leamer and productive member of society. * 

5) An aim of general education is to attain the skills in communication necessary 
for further growth as a lifespan-leamer and productive member of society. * 

6) An aim of general education is to attain the skills in quantification necessary 
for further growth as a lifespan-leamer and productive member of society. * 

7) An aim of general education is to attain the skills in synthesis necessary for 
further growth as a lifespan-leamer and productive member of society. 

It should be evident that there is no single procedure adequate to assess all of these outcomes. 
However, there is a core of general education competencies that can be identified as academic 
skills and which can be assessed using nationally standardized tests. These skills are analysis, 
communication, and quantification, and they are marked with *'s above. We will return to 
operationalizing this definition of general education later (see page 30). Continuing with defining 
the project, one moves to consider identification of the population. 

At many colleges and universities one can clearly identify students as admitted to the college or 
program, graduating or finishing a program, etc. Some of these same criteria are used at College 
of DuPage when students in programs are clearly admitted and completed. Such is not, 
however, the situation with general education. Students enroll for a few oourses, come to 
complete degrees, or come to earn enough credit to transfer. Students move on, transfer, stop- 
out, and are satisfied based on their own criteria. Thus, the college has no controls that would 
allow us to identify and encourage exit testing or assessment. 



Along with this lack of dearly identified qualification, past efforts in the late 1980's by the college 
to recruit students' partidpation in testing have failed despite rewards induding free course work 
and bookstore coupons. (It may be that the idea of testing is the issue, given students' 
willingness to partidpate in focus groups when offered pizza and pop.) 
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Responses to the assessment challenges 



Designing a response to these challenges resulted in applying a process first developed by the 
Value Added Committee in 1986. It was that committee which first designed a random section- 
based sampling used to field-test the six CAAP area-tests for ACT. 

Why the CAAP tests? 

Since one goal of general education assessment was to provide a comparison to other 
institutions, selection from among three national standardized tests of general skills was 
undertaken rather than the development of a local college-specific test of competence. Members 
of the Student Outcome Assessment Committee examined prospectuses from each of the three 
national tests. They selected the College Assessment of Academic Proficiency (CAAP) based on 
four observations. 

> CAAP provided national averages for both 2-year and 4-year public colleges for both 
first-year and sophomore levels. 

> CAAP provided six separate area-tests that could be aggregated as valid and reliable 
institution level measurements. This feature contrasted with more global single tests 
from which subject area scores were derived. 

> Each of these area tests could be administered in 50 minutes (according to the test 
protocol documents). 

> CAAP was judged least dependent on specific content ideas and, therefore, was 
judged most likely to evaluate general learning when students were not mandated 
into specific core courses. 

The six area-tests lined up well with a core of general education skills. The general education 
outcome of Analysis aligns with the CAAP area-tests of Critical Thinking and Science 
Reasoning. The general education outcome of Communication skills can be defined as 
including four skills - reading, writing, listening, and speaking. Two of these skills align with three 
CAAP area-tests. Writing can be assessed with the Writing Skills (a multiple choice test) and 
Essay Writing (a demonstrated essay) area-tests. After the first two years of testing, the Essay 
Writing area-test was suspended as described later (see page 38). The college level Reading 
area-test examines context reading in both the arts and social sciences. Assessment of 
Quantification aligns with the Mathematics area-test that covers material from algebra through 
calculus. 

Assessment using locally designed questions 

The CAAP instrument provided an additional resource, space for students to record nine locally 
designed questions. Two of these questions have been used to collect information about goals 
and prior college credits. Three of the questions focus on time commitments to study time, 
employment, and household duties. In 1999 one of the survey questions collected information 
about computer use. In 2001 a question was included that focused on advising and counseling 
services. For the past two years three of the questions have focused on assessment of some of 
the areas of general education not covered by the standardized tests. 

Assessment of the outcomes of understanding, appreciation and development of values in the 
areas of culture, environment and ethics, the first three outcomes listed in the general education 
statement, are not as easily implemented. Since the sampling method used to administer the 




3 



General Education Skills Development 



9 




CAAP provides a representative sampling of students, three of the institutional questions on the 
CAAP response form were designed to begin assessment of students' perceptions in these areas 
in 2000 and 2001. With the choice of an evaluation tool in place, the next issue was student- 
subject selection. 

Sampling model 

Because we concluded that mandatory college-wide testing was not feasible and voluntary 
testing efforts of the past resulted in high recruiting oosts for low participation, a sampling model 
utilizing a stratified, random sampling of class-sections was developed. The random selection of 
dass-sections assured testing a broad range of students with relatively low rates of non- 
partidpation. A high rate of partidpation is an important step in developing a representative 
sample. But, as may be evident and perhaps already distracting to the reader, such testing will 
result in the assessment of a range of students, only some of whom may match the ideal 
characteristics of entering first-year or completing, graduating or transferring sophomores. 
Utilizing this full range of students' data is an important issue we considered when designing the 
analysis of these data. The design of this all-encompassing analysis will be discussed later (see 
page 5). 

Assessment tests are conducted twice a year - fall and spring rounds. In each round of 
assessment we attempted to gather at least 100 tests in each subject area. Thus, approximately 
500 tests are administered in each of the current testing rounds. The model called for random 
sampling from all possible sections of introductory credit-dasses (100 level single classes and 
sequences dasses ending in 1 during the fell). During the spring the sampling was drawn from 
200 level courses and end of sequences courses. 

After initial identification of dass-sections, the faculty assigned to each section are contacted with 
a request for their cooperation. Sections are drawn from throughout the college and indude 
both full-time and part-time faculty at the Glen Ellyn campus and off-campus. Faculty 
cooperation has been very high with only an occasional refusal. Testing is scheduled for a 
speafic day and time, although it is suggested that students not be told the spedfic date, only a 
range of possible dates. To encourage student cooperation, faculty are provided handouts 
explaining CAAP assessment, which can be distributed to students. 

When CAAP tests are administered in the selected sections, each of the subject areas is 
rotational ly assigned to some students. Thus, in any one section some students complete each 
of the subject area-tests. 

Results from the first cyde of testing were analyzed and condusions reported in the fell of 1999. 1 
Since the sampling produced a reasonable number of cases and the analysis revealed no 
evidence of sampling bias, a second cyde of testing was completed in the fell of 1999 and spring 
of 2000 and a third in 2000-2001. The following report utilizes both of these cohorts. 2 



1 An Assessment Report on Students' General Education Development at College of DuPage. 
October 12, 1999. Student Outcome Assessment Committee, College of DuPage. 

2 In analyses not reported here the 2001 cohort was evaluated with regard to similarities and 
significant differences compared with the 1999 and 2000 cohorts. None of the differences noted 
were substantial or significant enough to warrant separate reporting. 
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Analysis model — Questions being addressed 

Given the challenges of defining and administering a pre-post test model at COD, this cross- 
section research design accommodates those constraints. If one aooepts that a dear pre-post 
test model cannot be implemented, but that a representative sampling of all students can be 
drawn using the data collection method described, then several approaches to analysis can be 
implemented. 

Although comparing a cross-section of students beginning their studies with another cohort near 
completion of their studies is a less desirable alternative than the pre-post model 3 , it is a 
necessity. These comparisons can focus on two questions. 

> Are the entering and leaving students similar to national standards for similar 
students at other colleges and universities? 

> Is there an indication that students have changed during their studies at an 
institution? 



In the "Comparison" chapter (see page 31) these types of models are reported. However, it 
needs to be noted that such comparisons are limited in the number of cases which can be 
classified as appropriate to the two points used in the analysis. In this round just over six 
hundred cases out of 2345 cases (26%) are classified as either entering first-year or completing 
sophomores. For the purpose of comparison a third category was defined to include those 
students at a mid-point in their studies. These students might be classified as end of first-year 
year or beginning of sophomore year. In this report they are labeled as mid-studies. The mid- 
studies classification contains an additional 26% of the cases. However, there are no national 
norms against which these students can be compared. 

In order to make more efficient use of the tests collected, a broader analysis lies in constructing 
a statistical model that uses cases at all points along the educational development continuum. 
One approach to doing this lies in modeling general education skills development based on a 
continuous development over the number and type of courses taken. Such a model can use 
most of the data collected in that it focuses on more than the initial and final points on the 
continuum of learning. Such models, although more complex than the two-point analysis, can be 
very interesting. These models are reported starting on page 43. 

Before beginning these comparisons, the characteristics of students taking one of the CAAP area- 
tests during the six rounds are summarized in the following chapter. These examinations focus 
on developing an understanding of students' characteristics. 



3 The major limit of a cross-sectional design is that any change among subjects in skills must be 
more substantial in a cross-sectional design than in a pre-post design in order to attain statistical 
significance. Thus, as in the case of general education, where change as subtle as one-half of 
the beginning standard deviation are observed, the power to detect significant change is reduced 
from that possible in a pre-post model. 
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General student characteristics 



As previously reported, the 1999 cohort consisted of 1148 cases. The 2000 cohort contained 
1191 cases. With the suspension of the use of the Essay area-test the 2001 cohort contained 
979 cases. In each of the cohorts a few cases could not be matched to student records in the 
student tracking system; these cases were removed from the data set. Table 1 reports the 
numbers of tests in each of the six subject-areas by the testing cohort 



• Table 1: Distribution of area tests by cohort 



Test Cohort 






Fall, 

1998 

(19991) 


Spring, 

1999 

(19993) 


Fall, 

1999 

(20001) 


Spring, 

2000 

(20003) 


Fall, 

2000 

(20011) 


Spring, 

2001 

(20013) 


Total 


Writing Skills 


Count 
% Test Cohort 


112 

18.5% 


85 

15.7% 


88 

16.9% 


115 

17.1% 


92 

19.4% 


108 

21.4% 


600 

18.1% 


Math 


Count 
% Test Cohort 


96 

15.8% 


93 

17.2% 


90 

17.2% 


114 

16.9% 


95 

20.0% 


94 

18.7% 


582 

17.5% 


Reading 


Count 
% Test Cohort 


99 

16.3% 


96 

17.7% 


86 

16.5% 


109 

16.2% 


96 

20.2% 


102 

20.2% 


588 

17.7% 


Critical Thinking Count 

% Test Cohort 


98 

16.2% 


91 

16.8% 


86 

16.5% 


117 

17.4% 


97 

20.4% 


105 

20.8% 


594 

17.9% 


Science Reasoning Count 
% Test Cohort 


96 

15.8% 


93 

17.2% 


86 

16.5% 


112 

16.6% 


95 

20.0% 


95 

18.8% 


577 

17.4% 


Essay 


Count 
% Test Cohort 


105 

17.3% 


84 

15.5% 


86 

16.5% 


106 

15.8% 






381 

11.5% 


Total 


Count 
% Test Cohort 


606 

100.0% 


542 

100.0% 


522 

100.0% 


673 

100.0% 


475 

100.0% 


504 

100.0% 


3322 

100.0% 



Each of the cohorts represents a reasonable distribution of the area-tests. The sampling 
methods and the high rates of participation and effort reported later (see page 12) support a 
conclusion that these data are a solid and representative sampling of the college's student 
population. When these data are combined with information retrieved from the student tracking 
system, we have a rich and broad based view of students' general education skills. 

These 3322 cases represent students across a spectrum of characteristics. In order to classify 
cases representative of target cohorts, information from the student tracking system is combined 
with the test results. For the purpose of comparison with the averages reported by ACT and 
comparison of entering first-year students with mid-studies and completing sophomores, three 
classification of students are made (see page 29 for a more complete explanation of this 
process). The following summarizing tables divide cases into two categories, those not selected 
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as representative of these three points and those selected as representative of the three points in 
a student's studies. 

Table 2 summarizes the sex distribution of the sample. Fifty-six percent (56%) of the 
participants are female. There is no significant difference in the distribution between those in 
the selected and not selected cohorts. 

• Table 2: Sex by selection 



Frequency Percent Valid Percent 



Not Selected 


Valid 

Missing 

Total 


Male 

Female 

Total 

Not reported 


691 

867 

1558 

7 

1565 


44.2 

55.4 

99.6 

.4 

100.0 


44.4 

55.6 

100.0 


Selected 


Valid 


Male 


768 


43.7 


44.0 






Female 


977 


55.6 


56.0 






Total 


1745 


99.3 


100.0 




Missing 


Not reported 


12 


.7 






Total 




1757 


100.0 





The average age among cases selected is 22 years old with a range of 15.7 to 97 years old 
(s.d.=.78). Among cases not selected, the average age is 26 years old with a range of 16.8 to 
94.4 years old. Cases were selected in part based on students who either began their studies at 
COD or cases that self-reported having earned fewer than 20 quarter-hour credits prior to 
attending COD. Thus, students who have college experiences from a variety of schools are less 
likely to be in the selected cohort and more likely to be older. Among entering first-year 
students, the selection criteria required enrollment during the next testing round in order to 
control for stop-out bias when compared with completing sophomores. 

Table 3 lists the self-reported ethnic/racial categories of the participants. Eighty percent (80%) 
of the selected participants classified themselves as "white, Caucasian." In contrast seventy- 
three percent (73%) of the "Not selected" participants classified themselves in a similar category, 
The "Asian/Pacific Islander " category was the second largest classification at 8% among the 
selected cases and 12% among the not selected cases. Once again these differences reflect the 
life-history of the participants. Issues of race and ethnicity are complex interactions to analyze. 
Since it is our purpose to try to identify the impact of COD's educational processes and 
statistically control for other external factors, the selection of cases with the fewest external 
educational experiences appears warranted. It should be noted that all of the cases are used in 
the educational models reported later in this report (see page 43). 
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• Table 3: Ethnic self identification by selection 







Frequency 


Percent 


Valid Percent 


Not Selected Valid 


Black/African - American 


43 


2.7 


3.1 




Amer. Indian/ Alaskan Native 


4 


.3 


.3 




White Caucasian 


1021 


65.2 


73.1 




Mexican-American / Chicano 


54 


3.5 


3.9 




Asian/ Pacific Islander 


166 


10.6 


11.9 




Puerto Rican/ Cuban/ Hispanic 


32 


2.0 


2.3 




Filipino 


32 


2.0 


2.3 




Other 


44 


2.9 


3.2 




Total 


1396 


89.2 


100.0 


Missing 


System missing 


88 


5.6 






Prefer not to respond 


81 


5.2 






Total 


143 


10.4 




Total 




1565 


100.0 




Selected Valid 


Black/African - American 


32 


1.8 


2.0 




Amer. Indian/ Alaskan Native 


3 


.2 


.2 




White Caucasian 


1286 


73.2 


80.0 




Mexican-American / Chicano 


54 


3.1 


3.4 




Asian/ Pacific Islander 


129 


7.3 


8.0 




Puerto Rican/ Cuban/ Hispanic 


28 


1.6 


1.7 




Filipino 


28 


1.6 


1.7 




Other 


47 


2.7 


2.9 




Total 


1607 


91.5 


100.0 


Missing 


System missing 


54 


3.1 






Prefer not to respond 


96 


5.5 






Total 


150 


8.5 




Total 




1757 


100.0 





The 14.2% non-native English speakers in the COD sample are substantially higher than the 
3.8% and 2.9% non-native speaks found in the CAAP User Norms for first-year and sophomores 
in two-year public colleges. Eight-seven percent (87%) of the selected participants identified 
English as their first language compared with eight-three percent (84%) of the not selected 
cohort. 

Seventy-five percent of the respondents reported being enrolled full-time during the quarter of 
their testing. Sixty-seven percent (66.6%) of the not selected cases and eighty-three percent 
(83.3%)of the selected cases reported full-time enrollment. Once again this difference is 
reflective of expectations for these cohorts. 

Finally, seventy-seven percent (77.4%) of the respondents reported beginning their college 
enrollment at COD. Among the selected cohort 90% reported beginning at COD. In contrast 
only 63.1% of the cases not selected reported beginning their college enrollment here. 

Table 4 lists the goals of participants. The leading goal, at 54%, is transfer to another college or 
university. The observation that this focus is greater than the 31% targeting an Associates 
Degree provides support for the observation that degree completion is not a valid criterion for 
evaluation of student outcomes. It should be noted that this percent of students who rank 
transfer as most important has been increasing over the past three years, while those ranking 
seeking a degree as most important has declined. Perhaps this reflects the change in emphasis 
from the compact structure for transfer to the I.A.A. options. 
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• Table 4: Most important goal 





Frequency 


Valid Percent 


Assotiates Degree 


922 


31.3 


Certificate in spedfic field 


147 


5.0 


Transfer to college or university 


1587 


53.8 


Transfer to technical school 


36 


1.2 


Upgrade current job skills 


42 


1.4 


Leam a spedfic skill 


59 


2.0 


Personal Interest 


68 


2.3 


Other 


88 


3.0 


Total 


2949 


100.0 



Table 5 summarized responses recorded in the student tracking system. Students self-reported 
both plans to transfer and plans to graduate. Thirty-five percent (35%) affirmed both goals. 
Sixty-seven percent (67%) reported plans to transfer and not graduate. Forty-three percent 
(43.4%) reported plans to graduate and not transfer. 



• Table 5: Plans to graduate and transfer 







Plan to transfer 








No 


Yes 


Total 


No 

Dl^n trh nraH i nko 


Count 
% of Total 


817 

24.7% 


1054 

31.9% 


1871 

56.6% 


r lan lu yrduuditi Ygs 


Count 
% of Total 


275 

8.3% 


1161 

35.1% 


1436 

43.4% 


Total 


Count 
% of Total 


1092 

33.0% 


2215 

67.0% 


3307 

100.0% 



Table 6 summarizes the self-reported number of hours per week spent studying outside of class 
for the respondent's average dass. Partidpants reported a mean average of 4.46 hours spent 
studying per week for their average dass (s.d.= 2.68). Among entering first-year students in the 
selected cohort, the average study time was 3.9 hours. This increased to 4.17 hours in mid- 
studies students and 5 hours among completing sophomores. Among the cases not selected, the 
average for first-year students is slightly higher. Mid-studies selected-students study slightly less 
than those not selected. But selected sophomore students study almost a half-hour more than 
sophomores not selected. None of these differences are statistically significant in their size. 

As a basis for comparison, most courses range from 3 to 5 hours. The plurality of courses is 5- 
quarter hours. Based on these reports and assumptions, it appears the average student spends 
about one hour outside of dass for each hour in dass. This is below the two-hour ratio 
suggested in both state statutes and general advising. The relationship of study time to general 
education skills is examined later in the modeling chapter on page 43. 



• Table 6: Hours studied per week in average course 







N 


Mean 


Std. 

Deviation 


Not Selected 


First-year 


226 


3.65 


2.50 




Mid-studies 


393 


4.53 


2.57 




Sophomores 


393 


4.53 


2.57 


Selected 


First-year 


421 


3.90 


2.69 




Mid-studies 


706 


4.17 


2.67 




Sophomores 


397 


5.05 


2.70 
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One time commitment that is frequently characterized as cutting into students' studies is 
employment. For this examination we included recognition that volunteer activities might be 
considered similar to employment This classification is not intended as a judgment of the worth 
of community services nor a judgment to exclude learning from community service from learning 
in an educational setting. 

Table 7 summarizes the time spent on employment (and volunteer activities) and household 
tasks. The percent of participants reporting no paid employment or no household work is five to 
seven percent lower in the combined sample than in the 1999 cohort. The median report for 
employment for selected entering first-year, mid-studies, and completing sophomores was the 
category 16 to 30 hours per week. The median report for household work for selected first-year, 
mid-studies, and completing sophomores was the category 1 to 15 hours. 



Table 7: Time commitments to employment (volunteering) and househol d work 





Employment (volunteer) 


Household 






Frequency 


Percent 


Frequency Percent 


No commitments to... 


441 


15.0 


325 


11.1 


1 -15 hours per week 


436 


14.9 


1874 


64.2 


16-30 


1127 


38.4 


390 


13.4 


31-45 


711 


24.2 


127 


4.4 


Over 45 hours per 
week 


219 


7.5 


203 


7.0 


Total 


2934 


100.0 


2919 


100.0 



Observations based on three cohorts 



Although comparison of the three test cohorts (1999, 2000, 2001) is not an issue being examined 
in this analysis, some observations need to be documented. In each of the cohorts the goal was 
to obtain at least 100 scores in each subject-area test (except Essay Writing) in each of the six 
testing cydes. Examination of Table 1: Distribution of area tests by cohort on page 7 documents 
that we came dose to this goal. 

Table 8 lists the area-test by educational level for the selected cases. It is important to note that 
once cases were selected, about one hundred first-year and completing sophomores were in 
each test-area cohort. The number of completing sophomores ranged from seventy-four to 
eighty-six cases. Thus, with the combination of the three years the numbers of cases utilized in 
the following analyses are at a level of acceptability for parametric statistical tests. 



• Table 8: Area-test by educational level 









Area-test 




Total 




Writing 






Critical 


Saence 






Educational Level 


Skills 


Math 


Reading 


Thinking Reasoning 


Essay 




First-year 


114 


110 


102 


123 


112 


103 


664 


Mid Freshmen-Sophomore 


134 


155 


145 


140 


134 


127 


835 


Completing Sophomores 


86 


74 


80 


86 


76 


44 


446 


Total 


334 


339 


327 


349 


322 


274 


1945 



Examinations and comparison of the three cohorts have also confirmed our expectation that the 
random stratified sampling of course sections produces an appropriate sample. However, there 
are non-biased but significant differences in some of the characteristics of these three cydes. 
Where these differences are of interest, they are reported with the assoaated observations. 
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The aggregating of these three cydes of testing into one set of cases for analysis has produced a 
sample that, as theory would predict, smoothes out the specific test cyde differences. Thus, the 
combined pool of participants has provided a basis for the following analysis that we condude is 
a strong basis for generalization. The question of whether future assessments should be limited 
to data from three years or indude more than three years will need to be addressed next year. 

At this time it appears that the number of cases is suffident to build strong inference, and that 
differences among the cohorts is minimal. However, as the project progresses it is expected that 
including too many different cohorts will weaken usefulness of the generalizations and 
condusions. 



Participation and motivation 



One issue of concern in considering student outcome assessment when measured in a low stakes 
setting, such as the CAAP testing at COD, is the impact of partidpation, motivation and effort by 
students on the measured outcomes. The sampling of dass sections used to construct a random 
sampling and the testing procedure of unannounced testing generate a very high partidpation 
rate. One check on this partidpation rate is a comparison of the number of students enrolled in 
the sampled course-sections and the number of test takers. We have not formally collected 
these data, but our estimation based on total numbers tested and totals enrolled is that 
partidpation rates are between 70% and 80% of the tenth day enrollment listings. Accepting 
that the tenth-day enrollments contain some students not actively pursuing studies, we conclude 
a very high level of partidpation and congruence between selection and partidpation. The next 
issue concerns the effort these subjects put into testing. 

Table 9 summarizes the self-report of students' efforts on the CAAP tests. The ACT Essay test 
form does not indude response space for the institutional questions. While we tried to collect 
student responses from those taking the Essay test on separate pages, collection was not 
consistent. So none of the responses from students completing the Essay area-test have been 
induded. Among student completing the other five subject-area tests, nineteen percent (19%) 
did not report their effort 



• Table 9: Self-reported effort 



Frequency Percent 


Gave no effort 


69 


2.9 


Gave little effort 


323 


13.7 


Gave moderate effort 938 


39.9 


Tried my best 


1022 


43.5 


Total 


2352 


100.0 



Eighty-four percent (84%) of the respondents reported that they either tried their best or gave 
moderate effort. The median ranking was "Gave moderate effort." This effort was consistent 
across the three rounds of testing. On several student characteristics there are significant 
differences in motivation. 

Among the cases selected for the comparison and value added analysis self-report of test effort 
increased with course work. First-year students reported 80.6% effort in the moderate and tried 
my best efforts. Mid-studies students reported 82% in these same two descriptions. Completing 
sophomores reported 86.8% in these two descriptions. These differences were significant 
(X 2 =14.263, sig.<.027). 
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Analysis of these self-reports indicates that there is a consistent and significant difference with 
fall participants reporting less effort than spring participants (Fall = 81% at "tried my best," and 
"gave moderate effort." Spring = 86% in these two categories. X 2 =30.3, sig. < .000). 

Examinations of these self-reports based on subject-area tests indicated a significant dependence 
of effort on the subject-area test (X 2 =86.47, sig. <.000). Greatest efforts were reported for 
writing skills (90%) and critical thinking (88%). Math (82%) and reading (81%) were next in 
effort. The lowest effort level for science reasoning was still a solid effort at 75% in these two 
categories. 

On average women (at 87%) reported significantly greater effort (X 2 =47.39, sig. <.000) than 
men (79%). 

Age was also significantly related to effort (X 2 =95.087, sig. < .000). The general pattern 
indicated increased efforts with increasing age. Seventy-eight percent (78%) of the participants 
younger than 18 reported in the first two categories compared with 82% in the 18-24 
classification. In each of the six-year categories from 25 to 55 the self-reports ranged from 85% 
to 100%). In the age category 61 and older effort reports fell to 58.3% for the two highest 
effort rankings. 

Analysis of the ethniq/race reports of effort indicated that those self-identifying with Filipino, 

Black African-American, White Caucasian, and Asian/Pacific Islander reported similar rates of 
87.8%, 85.5%, and 85% respectively. Mexican and Puerto Rican respondents reported 75% 
effort at the top two levels. The lowest ranking of effort came from those choosing "other" 
(73%). The "Native American" category contained too few cases to reliably report. These 
differences were significant (X 2 =39.492, sig. <. 009). 

There was no significant difference between the three test-cycle cohorts except the seasonal 
effect noted above. There was no significant difference in the efforts reported by students 
speaking English as their first language and those who did not speak English as their first 
language. 

During the spring 2001 test round the Student Outcomes Assessment Committee implemented a 
random award of one gift certificate for $10 in each of the class sections. Implementing this was 
not difficult. As a part of the preparation for testing, this random reward was announced. After 
each class test session an answer sheet was drawn at random and the sheet checked for self- 
reported effort in the top two categories and no evidence of patterned answering. The gift 
certificate was sent back to the cooperating teacher who delivered the certificate to the student, 
as she/he considered appropriate. Comparisons of the self-reported motivation from each of the 
spring rounds of testing indicate a consistent 3.3 ranking. Thus, there was no apparent impact 
of these gift certificates on the self-reported motivation. However, this is not to say that the 
acknowledgement as an indication of thanks is of no public relations importance. Given the low 
costs associated with this activity, it should continue for another round in Spring 2002. 

In conclusion, while there may be some minor evidence of "alienation," whether focused on 
hostility towards schooling or testing, overall the participation and motivation of respondents is 
considered appropriate and acceptable. 
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Chapter 



Development in cultural and 
environmental appreciation and 
ethical responsibility 



The implementation of CAAP testing was recognized as a starting point for assessing general 
education skills. It was accepted that the subject-area tests covered only a core of the outcomes 
targeted in the goal statement. The challenge of how to assess other areas of general education 
and the availability of locally designed questions on the CAAP form supported collecting a brief 
survey of a representative random sample. Starting with the 2000 cycle of assessment, three of 
the locally designed questions focused assessment on the impact of courses on enabling students 
to understand and appreciate their culture and environment and develop a system of personal 
values based on accepted ethics that lead to civic and social responsibility. While there may be 
more authentic measures than self-report, at this time the only indicators available for 
consideration of these general education areas are self-report responses. The following 
summaries are based on all of the cases collected. Thus the entering freshmen cohort contains 
both persisting students and stop-outs. These summaries can serve as a baseline for dialogue 
and future assessments of these areas of student acquisition of general education. 

The form of the questions was similar. 

> Based on your experiences at College of DuPage, how have your courses impacted 

your understanding and appreciation of your culture? 

> Based on your experiences at College of DuPage, how have your courses impacted 

your understanding and appreciation of the environment? 

> Based on your experiences at College of DuPage, how have your courses helped you 

develop personal values based on accepted ethics that lead to civic and 

social responsibilities? 

Responses for each of the questions were parallel. 

0. I've not completed enough courses to make a judgment. 

1. My COD education has had no impact on.... 

2. My COD education has had little impact on.... 

3. My COD education has had moderate impact.... 

4. My COD education has had meaningful impact.... 

5. My COD education has had very significant impact.... 



Appreciation of one's culture 



As with most general education outcomes, examination of development of values and attitudes 
over a series of courses is important. Table 10 lists the impact of course work on cultural 
appreciation by educational level. Nineteen percent (18.5%) of first-year students rank courses 
as having meaningful or very significant impact on their appreciation of culture. Among 
completing sophomores, 33.7% rank their COD courses' impact as meaningful or very significant 
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on development of a cultural appreciation. The data supports concluding that as students 
continue their studies at COD die percent of students who rate their COD courses as having 
meaningful or very significant impact on their cultural appreciation increases (X 2 =27.068, 
sig.c.001). 



• Table 10 Courses' impact on cultural appreciation by educational level 



Educational Level 


No 

Impact 


Little 

Impact 


Moderate Meaningful Significant 
Impact Impact Impact 


Total 


Entering First-year 


Count 


35 


29 


24 


15 


5 


108 


% within Education 


32.4% 


26.9% 


22.2% 


13.9% 


4.6% 


100.0% 


Mid-studies 


Count 


85 


66 


96 


52 


12 


311 


% within Education 


27.3% 


21.2% 


30.9% 


16.7% 


3.9% 


100.0% 


Completing Sophomores 


Count 


45 


53 


75 


63 


25 


261 


% within Education 


17.2% 


20.3% 


28.7% 


24.1% 


9.6% 


100.0% 


Total 


Count 


165 


148 


195 


130 


42 


680 


% within Education 


24.3% 


21.8% 


28.7% 


19.1% 


6.2% 


100.0% 



Appreciation of one's environment 



Table 11 lists the rankings from participants of their COD course's impacts on environmental 
appreciation. Thirty-four percent (33.6%) of first-year students rank their course work as having 
meaningful or very significant impact of appreciation of the environment. In contrast, thirty-one 
percent (30.5%) of the completing sophomores ranked their courses as having similar impact on 
their environmental appreciation. Among mid-studies students, rankings in these categories fall 
to 16.8%. Analysis supports concluding that course impact rating is not independent of 
educational level (X 2 =38.462, sig.000). Comparison of the percents in the lower rankings 
suggests that the there is a decrease in those ranking no or little impact and an increase in 
moderate impact while the higher ranking remains stable. 



• Table 11 Courses' impact on environmental appreciation 



Educational Level 


No 

Impact 


Little 

Impact 


Moderate Meaningful Significant 
Impact Impact Impact 


Total 


Entering First-year 


Count 


33 


18 


28 


35 


5 


119 


% within Education 


27.7% 


15.1% 


23.5% 


29.4% 


4.2% 


100.0% 


Mid-studies 


Count 


74 


79 


106 


41 


10 


310 


% within Education 


23.9% 


25.5% 


34.2% 


13.2% 


3.2% 


100.0% 


Completing Sophomores 

Count 


33 


68 


79 


63 


17 


260 


% within Education 


12.7% 


26.2% 


30.4% 


24.2% 


6.5% 


100.0% 


Total 


Count 


140 


165 


213 


139 


32 


689 


% within Education 


20.3% 


23.9% 


30.9% 


20.2% 


4.6% 


100.0% 
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Development of personal values based on accepted ethics 
that lead to civic and social responsibility 



Table 12 summarizes impact and educational level of rankings of participants concerning their 
evaluation of the impact of their COD courses on development of values based on accepted 
ethics that lead to dvic and sodal responsibility. Once again, impact is dependent on educational 
level indicating increasing impact over the course of students' studies (X 2 =30.935, sig.000). 
Although starting out as the lowest ranking of impact among first-year students (29.5%), the 
completing sophomore "meaningful" or "very significant" rankings are highest at 39.2%. 



• Table 12: Personal Values based on ethics 





No 

Impact 


Little 

Impact 


Moderate Meaningful 
Impact Impact 


Significant 

Impact 


Total 


Entering First-year 


Count 


32 


22 


37 


25 


13 


129 


% within Education 


24.8% 


17.1% 


28.7% 


19.4% 


10.1% 


100.0% 


Mid-studies 


Count 


57 


73 


124 


56 


14 


324 


% within Education 


17.6% 


22.5% 


38.3% 


17.3% 


4.3% 


100.0% 


Completing Sophomores 


Count 


37 


45 


75 


68 


33 


258 


% within Education 


14.3% 


17.4% 


29.1% 


26.4% 


12.8% 


100.0% 


Total 


Count 


126 


140 


236 


149 


60 


711 


% within Education 


17.7% 


19.7% 


33.2% 


21.0% 


8.4% 


100.0% 



Development of appreciation and values 



Examination of each of the three patterns of responses in the previous tables is oomplex. In 
each area the percent of respondents ranking their course experience as having "no" or "little" 
impact on appreciation and development in the area decreases substantially between first-year 
and completing sophomore students. In both the cultural and ethical development areas the 
percent ranking the impact at the higher level substantially increases between first-year and 
completing sophomore students. The pattern among mid-studies students is not consistent. The 
rise and fall of rankings brings to mind the observations that students enter oollege sure that 
there is a right answer and that their course work will deliver it. Then their studies lead them to 
enter a stage of increased skepticism and uncertainty prior to developing a complex and 
conditional logic of critical thinking. It's important to note that these observations may provide 
insight into the challenges of developing critical and complex thinking skills. 
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Shortly after the Student Outcomes Assessment Committee began its first round of general 
education skills testing in the Fall Quarter of 1998, the committee was challenged by faculty to 
look at student services within the institutional framework. One response to this challenge lies in 
the use of one of the nine locally designed questions on the CAAP form. The following question 
was inducted for the first time in the Fall 2000 testing cyde and repeated in the Spring 2001 
cyde. 



“How much have you used Advising and Counseling services, and what impact 
has that had on your college success?” 

The question comprises two issues - level of use and importance. Responses are reported 
using ten multiple<hoice codes. 

0) No enough contact to determine 

1) Minimal contact, not helpful 

2) Minimal contact, helpful 

3) Minimal contact, very helpful 

4 ) Moderate contact, not helpful 

5) Moderate contact, helpful 

6) Moderate contact, very helpful 

7) Substantial contact, not helpful 

8 ) Substantial contact, helpful 

9) Substantial contact, very helpful 



The question asks students to rank "advising and counseling" services. The two words carry 
different and significant connotations among those well informed about the college's organization 
and staffing. The goals and objectives of these two efforts represent important but separate 
efforts. Among "those in the know," the combining of these two components may be too high a 
level of aggregation. However, I doubt that students are aware or impacted by these 
organizational issues. I, therefore, posit that the question is a good summative look at the 
combined impact of this area of student service. 

Table 13 summarizes responses from the 979 respondents in this year's two testing cycles. Of 
these only 26 did not answer the question. 4 Given this very high level of response we judged 
that the question, despite its complex nature, was understandable and usable. 



4 The multiple component form of the question is more complex than a two-question series. The 
complex format has been a concern to some people because they think students may be having 
a hard time responding using the complex response matrix. It would be ideal to have this 
question broken out into two questions. However, given the limited number of survey response 
items, and the need to collect other information this is not possible. The high response rate of 
over 97% supports concluding that students did not have substantial problems understanding 
and responding to the question, although it may have taken them longer to decode than a two- 
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• Table 13: Reported advising and counseling contact and value 





Frequency 


Percent 


Valid Percent 


Not enough contact to determine 


280 


28.6 


29.4 


Minimal, not helpful 


142 


14.5 


14.9 


Minimal, helpful 


218 


22.3 


22.9 


Minimal, very helpful 


36 


3.7 


3.8 


Moderate, not helpful 


37 


3.8 


3.9 


Moderate, helpful 


137 


14.0 


14.4 


Moderate, very helpful 


38 


3.9 


4.0 


Substantial, not helpful 


17 


1.7 


1.8 


Substantial, helpful 


32 


3.3 


3.4 


Substantial, very helpful 


16 


1.6 


1.7 


Total 


953 


97.3 


100.0 


Missing 


26 


2.7 






979 


100.0 





The two-component format of the question is complex and a single listing of responses is of 
limited interpretive value. Therefore, responses were coded into two variables - advising and 
counseling level of contact, and value of advising and counseling among those who thought they 
have enough contact to make a judgment. 

Table 14 summarizes the student rankings of value by contact among the 673 students reporting 
enough contact to make a value judgment. These rankings represent a college wide average. 
The relationship between contact and value is statistically significant (X 2 = 34.193, sig. .000). 
Since the X 2 test evaluates categorical data, not linear trends, interpretation is limited to saying 
that within a table knowing the level of contact significantly predicts the probabilities of the 
associated value ranking. 

At all levels of contact, sixty-four (64.1%) to seventy-four percent (73.8%) of the respondents 
value that contact as helpful or very helpful. This strong endorsement should be the overall 
conclusion. Other areas that might be considered lie in following-up with those ranking the 
services as not helpful. Among those with minimal contact thirty-six percent (35.9%) see that 
contact as not helpful. This will probably lead to their not continuing that contact, although if 
they do they may improve their ranking of the services. Among those with substantial levels of 
contact twenty-six percent (26.2%) rank that contact as not helpful. This may be attributed to 
either un-realistic expectations or severity of issues that exceed the capacities of student 
advising/counseling programs. 



question format. A change in the visual formatting of a similar question is being attempted to 
see if such appearance cues may help reduce the response time. 
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• Table 14: Advising and counseling contact by value 







Minimal 

contact 


Moderate 

contact 


Substantial 

contact 


Total 


Not helpful Gount 


142 


37 


17 


196 




% Advising Worth 


72.4% 


18.9% 


8.7% 


100.0% 




% Advising Contact 


35.9% 


17.5% 


26.2% 


29.1% 


Helpful 


Count 


218 


137 


32 


387 




% Advising Worth 


56.3% 


35.4% 


8.3% 


100.0% 




% Advising Contact 


55.1% 


64.6% 


49.2% 


57.5% 


Very helpful Count 


36 


38 


16 


90 




% Advising Worth 


40.0% 


42.2% 


17.8% 


100.0% 




% Advising Contact 


9.1% 


17.9% 


24.6% 


13.4% 


Total 


Count 


396 


212 


65 


673 




% Advising Worth 


58.8% 


31.5% 


9.7% 


100.0% 




% Advising Contact 


100.0% 


100.0% 


100.0% 


100.0% 



Given the variety of students' goals, the first comparison presented in Table 15 focuses on a 
comparison of advising and counseling contacts by self-selected goals. The level of contact with 
these services is significantly related to these goals (X 2 = 40.020, sig. .007). Students with 
focused goals (i.e. upgrade a skill, leam specific skill and personal interest) have lower levels of 
contact with advising and counseling than do those with general goals (i.e. degree, certificate 
and transfer). The highest level of contact arises among those targeting transfer to a technical 
school (37.5%) and those seeking an Associate degree (32.1%) and transfer (29.7%). 



• Table 15: Goals by advising/counseling contact 





Not enough 


Minimal 


Moderate 


Substantial 






to rank 


contact 


contact 


contact 


Total 


Associates Degree Count 


74 


103 


63 


21 


261 


% Most Important Goal 


28.4% 


39.5% 


24.1% 


8.0% 


100.0% 


Certificate in specific field Count 


19 


21 


5 


5 


50 


% Most Important Goal 


38.0% 


42.0% 


10.0% 


10.0% 


100.0% 


Transfer to college universityCount 


135 


243 


127 


33 


538 


% Most Important Goal 


25.1% 


45.2% 


23.6% 


6.1% 


100.0% 


Transfer to technical school Count 


4 


6 


4 


2 


16 


% Most Important Goal 


25.0% 


37.5% 


25.0% 


12.5% 


100.0% 


Upgrade current job skills Count 


7 


4 


2 




13 


% Most Important Goal 


53.8% 


30.8% 


15.4% 




100.0% 


Leam a specific skill Count 


9 


6 


3 


1 


19 


% Most Important Goal 


47.4% 


31.6% 


15.8% 


5.3% 


100.0% 


Personal Interest Count 


14 


6 


3 


1 


24 


% Most Important Goal 


58.3% 


25.0% 


12.5% 


4.2% 


100.0% 


Other Count 


16 


7 


5 


1 


29 


% Most Important Goal 


55.2% 


24.1% 


17.2% 


3.4% 


100.0% 


Total Count 


278 


396 


212 


64 


950 


% Most Important Goal 


29.3% 


41.7% 


22.3% 


6.7% 


100.0% 



Table 16 focuses on the self-reported value of advising and counseling contacts by the students' 
goals. In this examination there is no significant relationship between their goals and the value 
they perceive in their contact with these services (X 2 = 14.786, sig. .393). 
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• Table 16: Advising/oounseling value by goal 





Not helpful 


Helpful 


Very helpful 


Total 


Associates Degree Count 


46 


107 


34 


187 


% Most Important Goal 


24.6% 


57.2% 


18.2% 


100.0% 


Certificate in specific field Count 


14 


13 


4 


31 


% Most Important Goal 


45.2% 


41.9% 


12.9% 


100.0% 


T ransfer to col lege or university Count 


119 


237 


47 


403 


% Most Important Goal 


29.5% 


58.8% 


11.7% 


100.0% 


T ransfer to technical school Count 


4 


5 


3 


12 


% Most Important Goal 


33.3% 


41.7% 


25.0% 


100.0% 


Upgrade current job skills Count 


2 


4 




6 


% Most Important Goal 


33.3% 


66.7% 




100.0% 


Learn a specific skill Count 


3 


6 


1 


10 


% Most Important Goal 


30.0% 


60.0% 


10.0% 


100.0% 


Personal Interest Count 


3 


7 




10 


% Most Important Goal 


30.0% 


70.0% 




100.0% 


Other Count 


5 


7 


1 


13 


% Most Important Goal 


38.5% 


53.8% 


7.7% 


100.0% 


Total Count 


196 


386 


90 


672 


% Most Important Goal 


29.2% 


57.4% 


13.4% 


100.0% 



In addition to the primary goal question on the CAAP form, the student tracking system contains 
two separate student self-reports from contact at registration on plans to graduate and plans to 
transfer. The following four examinations utilize these two variables. 

Table 17 summarizes the levels of contact with advising and counseling by plans to graduate. 
Although there are slightly higher levels of contact among those planning to graduate, there is no 
significant relationship (X 2 = 5.790, sig. .122). 



» Table 17: Advising/counseling contact by plans to graduate 







No 


Yes 


Total 


Not enough to rank 


Count 

% Plan to graduate 


167 

32.4% 


109 

25.5% 


276 

29.2% 


Minimal contact 


Count 

% Plan to graduate 


209 

40.5% 


186 

43.5% 


395 

41.8% 


Moderate contact 


Count 

% Plan to graduate 


106 

20.5% 


104 

24.3% 


210 

22.2% 


Substantial contact 


Count 

% Plan to graduate 


34 

6.6% 


29 

6.8% 


63 

6.7% 


Total 


Count 

% Plan to graduate 


516 

100.0% 


428 

100.0% 


944 

100.0% 



Table 18 summarizes evaluations of advising and counseling by plans to transfer. Although those 
who plan to transfer run 31% with moderate to substantial contact compared with 23.2% for 
those who do not plan to transfer, these differences are not generally aocepted as significant 
given the 944 cases (X 2 = 7.193, sig. .066). 
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• Table 18: Advising/counselinq contact by plans to transfer 







N 


Y 


Total 


Not enough to rank 


Count 

% Plan to transfer 


90 

33.2% 


186 

27.6% 


276 

29.2% 


Minimal contact 


Count 

% Plan to transfer 


118 

43.5% 


277 

41.2% 


395 

41.8% 


Moderate contact 


Count 


51 


159 


210 




% Plan to transfer 


18.8% 


23.6% 


22.2% 


Substantial contact 


Count 


12 


51 


63 




% Plan to transfer 


4.4% 


7.6% 


6.7% 




Count 


271 


673 


944 


Total 


% Plan to transfer 


100.0% 


100.0% 


100.0% 



A similar pattern emerges in the relationship between evaluations and plans to graduate or plans 
to transfer. In Table 19 evaluations are not significantly different among those who plan to 
graduate and those who do not (X 2 = 1.308, sig. .520). 



» Table 19: Advising/counselinq by plans to graduate 







No 


Yes 


Total 


Not helpful 


Count 

% Plan to graduate 


103 

29.5% 


92 

28.8% 


195 

29.2% 


Helpful 


Count 

% Plan to graduate 


204 

58.5% 


179 

56.1% 


383 

57.3% 


Very helpful 


Count 

% Plan to graduate 


42 

12.0% 


48 

15.0% 


90 

13.5% 


Total 


Count 

% Plan to graduate 


349 

100.0% 


319 

100.0% 


668 

100.0% 



In Table 20 evaluations are not significantly different among those who plan to transfer and 
those who do not (X 2 = .745, sig. .689). 



• Table 20: Advisinq/oounselinq value by plans to transfer 







No 


Yes 


Total 


Not helpful 


Count 

% Plan to transfer 


54 

29.8% 


141 

29.0% 


195 

29.2% 


Helpful 


Count 

% Plan to transfer 


106 

58.6% 


277 

56.9% 


383 

57.3% 


Very helpful 


Count 

% Plan to transfer 


21 

11.6% 


69 

14.2% 


90 

13.5% 


Total 


Count 

% Plan to transfer 


181 

100.0% 


487 

100.0% 


668 

100.0% 



Table 21 summarizes advising and counseling contact by sex. There is a significant difference 
between males and females contact level (X 2 = 7.302, sig. .063). Males report a slightly higher 
rate of "not enough contact." Although males have somewhat higher rates of "minimal contact" 
and "substantial contact than women these differences are not substantial. The female rate for 
"moderate contact" is substantially higher than the male rate. 
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• Table 21: Advising/counseling contact by sex 







Sex 

Male Female 


Total 


Not enough to rank 


Count 
% Sex oode 


140 

31.5% 


140 

27.5% 


280 

29.4% 


Minimal contact 


Count 


189 


207 


396 




% Sex code 


42.6% 


40.7% 


41.6% 


Moderate contact 


Count 


82 


130 


212 




% Sex code 


18.5% 


25.5% 


22.2% 


Substantial contact 


Count 


33 


32 


65 




% Sex code 


7.4% 


6.3% 


6.8% 


Total 


Count 


444 


509 


953 




% Sex code 100.0% 100.0% 


100.0% 



Table 22 summarizes the value of these contacts by sex. There is no significant difference in 
these rankings between males and females (X 2 = .358, sig. .836). 



• Table 22: Advising/counselinq value by sex 







Sex 

Male Female 


Total 


Not helpful 


Count 
% Sex code 


91 105 

29.9% 28.5% 


196 

29.1% 


Helpful 


Count 
% Sex code 


171 216 

56.3% 58.5% 


387 

57.5% 


Very helpful 


Count 
% Sex code 


42 48 

13.8% 13.0% 


90 

13.4% 


Total 


Count 304 369 

% Sex code 100.0% 100.0% 


673 

100.0% 



The CAAP instrument provides seven racial and ethnic categories for self-designation by the 
respondent. Utilizing seven categories to examine contact and value is not effective since many 
of the non-white categories contain only a few cases. Aggregating several of the categories 
results in data that can be considered. Both the "African-American/Black" and "white/Caucasian" 
categories were retained. Respondents selecting the categories of Native American, Other, and 
Prefer not to respond were assigned to missing data. The Chicano and Hispanic categories were 
collapsed into one category. In a similar way the Filipinos and Asian Pacific Islanders were 
combined. This recoding resulted in four radal/ethnic categories. The level of advising and 
counseling servioe contact by these four categories is reported in Table 23. Racial and ethnic 
categories do not have a significant relationship with the level of contact (X 2 = 13.388, sig. .146). 





OCTOBER 16, 2001 



27 



• Table 23: Advising/counseling contract by ethnic category 







Not enough 


Minimal 


Moderate 


Substantial 








to rank 


Contact 


Contact 


Contact 


Total 


White, Caucasian 


Count 


196 


270 


140 


36 


642 


% Ethnic Category 


30.5% 


42.1% 


21.8% 


5.6% 


100.0% 


Black/African - American 


Count 


8 


7 


7 


4 


26 


% Ethnic Category 


30.8% 


26.9% 


26.9% 


15.4% 


100.0% 


Mexican-American, 
Chicano, Puerto Rican, 
Cuban, Hispanic 


Count 


11 


20 


14 


6 


51 


% Ethnic Category 


21.6% 


39.2% 


27.5% 


11.8% 


100.0% 


Asian/ Pacific Islander, 
Filipino 


Count 


26 


50 


30 


11 


117 


% Ethnic Category 


22.2% 


42.7% 


25.6% 


9.4% 


100.0% 


Total 


Count 


241 


347 


191 


57 


836 


% Ethnic Category 


28.8% 


41.5% 


22.8% 


6.8% 


100.0% 



When examining the relationship of ethnic and racial categories with value a conclusion is less 
certain. Table 24 summarizes value by ethnic racial group. The relationship of raoe-ethnicity to 
value is weak, and is not independent (X 2 = 10.986, sign. .089). Given that this relationship is 
based on 595 cases the relationship would generally be judged as not significant. However, it 
may be intuited that there may be a weak relationship between these categories and their 
associated values. 

• Table 24: Advising/counseling value by ethnic category 



Advising Counseling Worth 





Not helpful 


Helpful 


Very Helpful 


Total 


White, Caucasian Count 


137 


255 


54 


446 


% Ethnic Category 


30.7% 


57.2% 


12.1% 


100.0% 


Black/African - American Count 


4 


8 


6 


18 


% Ethnic Category 


22.2% 


44.4% 


33.3% 


100.0% 


Mexican-American, Chicano, 










Puerto Rican, Cuban, 










Hispanic Count 


14 


19 


7 


40 


% Ethnic Category 


35.0% 


47.5% 


17.5% 


100.0% 


Asian/ Pacific Islander, 










Filipino Count 


20 


58 


13 


91 


% Ethnic Category 


22.0% 


63.7% 


14.3% 


100.0% 


Total Count 


175 


340 


80 


595 


% Ethnic Category 


29.4% 


57.1% 


13.4% 


100.0% 



One student characteristic that might impact both use and worth of advising and counseling is 
language. The CAAP instrument provides a spaoe for students to self-identify their English 
language background. Table 25 summarizes advising and counseling contact for native speakers 
of English and those for whom English is not their first language. Native speakers of English are 
less likely to have contact with or have lower levels of contact with advising and counseling than 
non-native speakers of English (X 2 = 15.863, sig. .001). 
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• Table 25: English as 


first language by advising/counseling contact 




English First Language 


Total 




No 


Yes 




Not enough to rank Count 


31 


249 


280 


% English First Language 


21.2% 


31.0% 


29.5% 


Minimal contact Count 


55 


338 


393 


% English First Language 


37.7% 


42.1% 


41.5% 


Moderate contact Count 


42 


169 


211 


% English First Language 


28.8% 


21.1% 


22.3% 


Substantial contact Count 


18 


46 


64 


% English First Language 


12.3% 


5.7% 


6.8% 


Total Count 


146 


802 


948 


% English First Language 


100.0% 


100.0% 


100.0% 


Table 26 summarizes the evaluations of native and non-native speakers of English. Although 


non-native speakers of English report more contact with advising and counseling, their evaluation 
of these services are not significantly different from native English speakers (X 2 = 4.175, sig. 


.124). 








• Table 26: English as 


first language by advising/counseling value 




English First Language 






No 


Yes 


Total 


Not helpful Count 


30 


166 


196 


% English First Language 


26.1% 


30.0% 


29.3% 


Helpful Count 


63 


320 


383 


% English First Language 


54.8% 


57.9% 


57.3% 


Very helpful Count 


22 


67 


89 


% English First Language 


19.1% 


12.1% 


13.3% 


Total Count 


115 


553 


668 


% English First Language 


100.0% 


100.0% 


100.0% 


While these institutional summaries may provide a baseline indication of contact and value, it 



does not help us understand some of the underlying process. The following analyses utilize the 
three classifications of educational levels (see page 29). Table 27 summarizes the entering first- 
year, mid-studies, and completing sophomores rankings of advising and counseling services. 



Comparisons of level of contact and impact among educational levels 



The first and most striking observation is that over 45% of the entering first-year students have 
not had enough contact to rank these services. While this lack of contact declines significantly, it 
should be noted that these rankings were made during the fifth and sixth week of a term. By the 
time students are completing their COD learning, forty-six percent (46%) have had moderate or 
substantial contact with advising and counseling. And all but seventeen percent (17%) have had 
at least minimal contact. 
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• Table 27: Educational level by advising/counseling contact 

Entering Completing 







First-year Mid-Studies Sophomores 


Total 


Not enough contact to rank Count 

% Classified Educational Level 


83 

45.6% 


26 

22.0% 


20 

17.4% 


129 

31.1% 


Minimal contact 


Count 


60 


62 


42 


164 


% Classified Educational Level 


33.0% 


52.5% 


36.5% 


39.5% 


Moderate contact 


Count 


33 


23 


41 


97 


% Classified Educational Level 


18.1% 


19.5% 


35.7% 


23.4% 


Substantial contact 


Count 


6 


7 


12 


25 


% Classified Educational Level 


3.3% 


5.9% 


10.4% 


6.0% 


Total 


Count 


182 


118 


115 


415 


% Classified Educational Level 


100.0% 


ioo:o% 


100.0% 


100.0% 



Table 28 lists the values rankings of advising and counseling by educational ategories. Between 
sixty-three (63%) and seventy-four percent (74%) of these students find counseling and advising 
services either helpful or very helpful. A pattern that may be worth examining further is the 
increasing percent of completing sophomores who rank these services at "not helpful." As the 
pattern indicates that as students complete their studies some raise their evaluation while a 
larger portion lower their rankings of these support services. This may be due to changes in the 
issues faced by students as they near completion. Or it may be due to frustrations when their 
expectations that someone else will provide guaranteed answers are unmet. 



• Table 28: Educational level by advising/counseling value 







Entering 

First-year 


Mid-studies 


Completing 

Sophomores 




Not helpful Count 

% Classified Educational Level 


16 

16.2% 


26 

28.3% 


35 

36.8% 


77 

26.9% 


Helpful 


Count 

% Classified Educational Level 


69 

69.7% 


57 

62.0% 


46 

48.4% 


172 

60.1% 


Very helpful Count 

% Classified Educational Level 


14 

14.1% 


9 

9.8% 


14 

14.7% 


37 

12.9% 


Total 


Count 


99 


92 


95 


286 




% Classified Educational Level 


100.0% 


100.0% 


100.0% 


100.0% 



Given the previous observation concerning the 45.6% of first-year students reporting not enough 
contact to rank these services. Table 29 examines these first-year students with reference to 
their persistence or stop-out behavior. It is important to keep in mind the observation that stop- 
out does not mean these students did not transfer mid-year to other institutions. While there is 
no significant difference in the level of contact between these two classifications (X 2 = 2.897, sig. 
.408), persisters have a lower level of no contact and higher level of minimal contact than stop- 
outs. Thus, it appears that first-year students' outcomes in terms of persistence may be 
influenced by their contact with these student services or the contact level may reflect 
individuals' commitment and involvement. Other first-year research would lend credence to the 
contact and a road to college success. 
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• T able 29: First-year outcome by advising/counseling contact 

Native first-year outcomes 

Stop-out Persist Total 

Not enough to rank Count 23 54 77 

% Native first-year outcomes 53.5% 43.5% 46.1% 

Minimal contact Count 9 43 52 

% Native first-year outcomes 20.9% 34.7% 31.1% 

Moderate contact Count 9 23 32 

% Native first-year outcomes 20.9% 18.5% 19.2% 

Substantial contact Count 2 4 6 

% Native first-year outcomes 4.7% 3.2% 3.6% 

Total Count 43 124 167 

% Native first-year outcomes 100.0% 100.0% 100.0% 
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In this chapter we utilize two approaches in examining the aoquisition of general education skills 
at an institutional level. One approach utilizes comparison of entering first-year and completing 
sophomores to national averages for 2-year public community college students. This comparison 
is expanded with comparison with 4-year public university averages. A second approach utilizes 
comparisons among College of DuPage's entering first-year students, mid-studies, and 
completing sophomores. 

These comparisons require classification of cases based on students' course completion at COD 
and other colleges. This classification is necessary in order to identify learning outcomes 
resulting from COD experiences, not skills acquired at other colleges. This may appear to be an 
easy task, but it's not. Two specific issues need to be considered in order to control for 
identifiable biases. The general process is to use information from the student responses and 
student-tracking system (STS) to identify "ideal" COD cases, that is cases that would be typical of 
various stages in a student's studies if we were able to collect longitudinal developmental 
measures. 

One issue of concern in cross-sectional comparisons such as these is the impact of attrition or 
stopping-out. The research design applied to control for the first-year stop-out effect utilizes 
enrollment information from the STS for students' subsequent enrollment in the next testing 
round. Entering first-year students taking a CAAP testing in the fall round must be enrolled in the 
following spring round of testing in order to be classified as persisting. In a similar process those 
entering first-year students tested in the spring must be enrolled in the following fall in order to 
be classified as persisting. Those entering first-year students who are not enrolled in the 
subsequent testing round are classified as stop-outs. A traditional assumption, perhaps incorrect, 
might be that stop-outs may be primarily lower achievers. If this (or the opposite) is the case, 
then a bias is introduced in comparing entering cross-sections with persisting cross-sections. 

Such a bias might be labeled as a "freshmen survival effect." To avoid this bias, in the 
comparisons that follow persisting first-year students are compared with mid-studies, and 
completing sophomores. Specific comparisons of persisting students with those stopping-out are 
presented later, but as a general observation the differences are not substantial and sometimes 
in a direction opposite from expectations (see page 42). 

A second issue is that arising from the impact of reverse transfer. Determining the level of 
college studies for the native students beginning studies at COD is difficulty using the STS. For 
students who begin at other colleges and universities, there are several sources that may be 
used as a basis for this classification. In general students with few credits from other institutions 
may be classified into the comparison cohorts, while those with higher numbers of credits and 
degrees are excluded from these comparisons but included in the model building examinations 
starting on page 43. 

The process of classification for all students is similar. First, students self-reported on the CAAP 
answer sheet their educational level as freshman, sophomore, junior, senior, and other. 

Students also self-reported the number of credits earned and the number of credits earned at 
other colleges. The student tracking system (STS) is a second major source of information. 
Several indicators in the STS are used. Two indicators of educational level were retrieved from 
STS. First, the total credit hours earned at COD in college-level courses (=>100) at the end of 




General Education Skills Development 



32 



29 




the quarter during which testing occurred were identified. Second, utilizing transcripts (from the 
student tracking system) the cumulative number of college-level courses (=>100) successfully 
completed in each of eight subject-discipline areas and the total number of courses successfully 
completed were tallied. In addition to COD experiences, students' self-report of credits and 
degrees completed at other institutions need to be considered. 

The process of classifying student educational level involved utilization of all of these measures. 
One might hope that all of these measures might have general congruence for all of the cases. 
Unfortunately, some tend not to agree. To resolve conflicts the two measures from STS were 
considered more reliable than student self-reports. 

Educational level categories for analysis were defined as 1) entering persisting first-year 
students, 2) mid-studies - students between entry and completion, and 3) completing 
sophomore. To operationalize these definitions, the distribution of credits earned at COD and at 
other colleges was examined. Based on these examinations the following definitions were 
established. 

Entering persisting first-year (1) earned 20 or fewer quarter hours at the end of the quarter 
during which they completed the CAAP test. Further, in these comparison entering first-year 
students were included in analysis only if they self-reported beginning their studies at COD 
(classified as a "native-student") and if they persisted to the next testing round. 

Mid-studies (first-year/sophomores) (2) earned between 41 and 60-quarter hours at the 
end of the quarter during which they completed the CAAP test. Classification of mid-studies 
included students who reported earning fewer than 21 quarter-hour credits at other college or 
universities. 5 

Completing sophomores (3) earned between 81 and 110-quarter-hours 6 at the end of the 
quarter during which they completed the CAAP test. This classification also included students 
who reported earning fewer than 21 quarter-hour credits at other colleges and universities. 

This classification system is effectively applied to native-students (self-reported that their studies 
began at COD), although even here there are a number of students who self-report a different 
classification than that identified through the STS. This classification procedure also worked for 
those cases who reported earning very few credits at other schools but who did not begin their 
studies at COD. This classification scheme resulted in educational levels 1, 2, and 3 for those 
meeting the criteria specified above and 1.5, 2.5 for those cases with few credits from other 
colleges whose COD credits ranged between the three classifications defined. 

A greater challenge is classifying reverse transfer students who transfer in more than 20 quarter- 
hours credit. For those cases that were not native-students, several classification routines were 



5 Including non-native cases reporting few transfer credits was based on a comparison of these 
students reported in An Assessment Report on Students' General Education Development at 
College of DuPage, October 12, 1999. 

6 The upper limit defined in classification of educational level was identified during previous 
research. It was noted at that time that the sampling method included a few "perpetual" 
students earning credits well beyond the 96 quarter hour credits required for graduation and/ or 
transfer. Since the focus of this research is on the acquisition of general education skills, I 
judged that the few cases with course credits beyond 110 quarter-hours were outliers and were 
not representative of the general student population. Thus, these cases were not used in making 
comparative evaluations. 
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applied. The self-report of credits earned at other colleges were factored into an estimation of 
total credits. These projections resulted in educational levels ending in decimal fractions. 

Thus, in the following analysis, selection of cases with educational levels of 1, 2, or 3 provided 
dear categories of students meeting speafied criteria. These cases are most representative of 
COD learning outcomes. This selection was used in all of the following comparisons. The 
remaining cases with deamal educational levels were added to the selected cases for some of 
the descriptive analyses and the modeling which follows on page 43. 



COD and national norms 



One of the reasons for selecting ACT's Collegiate Assessment of Academic Proficiency (CAAP) was 
to provide a reference norm for comparison with COD outcomes. The annual publication of 
standardized norms provides such a reference. In the case of this analysis the Fall, 2000 CAAP 
User Norms are referenced. 7 

Table 30 through Table 35 list the mean-average and other statistical information for each of the 
three dassifications of educational level on one of the subject-area tests. Also listed on each 
table are the ACT-CAAP 2-year public college and 4-year public college national averages. As 
with any comparison of average drawn from samples, one must consider a margin of error before 
conduding significant differences. 

In order to compare the COD averages with those from one of the reference groups, a 95% 
Confidence Interval has been calculated. The Confidence Interval is a range of scores 
representing the upper and lower bounds that the mean-average might be if the entire 
population of students were tested rather than sampled. In other words, in the reading skills 
area-test, we can be 95% certain that the entering COD first-year student average would fell 
between 58.73 and 60.8. In a similar manner, we are 95% certain that the freshman 2-year 
public college average is between 59.36 and 59.6. Since the confidence interval for COD 
entering first-year students overlaps with the confidence interval for 2-year public college 
freshmen, we can condude that the difference between the COD average of 59.76 and the 2- 
year public college average of 59.6 is not statistically significant. 

This procedure has been applied to each of the tables and the observations are summarized prior 
to the supporting table. 

Reading skills 

Data from Table 30 indicates that entering COD first-year students are similar to the average of 
2-year public college freshmen. Entering first-year students are significantly below first-year 4- 
year public college freshmen. Completing COD sophomores have improved the rank of reading 
skills in a manner similar to the 2-year public college sophomore cohort. Although their average 
is still below the 4-year public college sophomore norm, the COD confidence interval is not 
statistically lower than the 4-year public college cohort. Given the increase across the three- 
measurement points, it is reasonable to condude that reading skills increase across COD course 



7 The CAAP User Norms: Fall, 2000 are based on the average of the previous three years' test 
takers who self-identified into the categories described in the published dassification categories. 
Thus, the averages we used were a general summary of students reporting first-year and 
sophomore status at 2-year and 4-year public colleges. In general these averages would tend to 
describe entering first-year and completing sophomores, but it should be noted that the averages 
are not criterion based. 



General Education 1 c Development 



31 




of study. ODD students show an increase exceeding the 4-year public cohort change. This may 
indicate that there is an upper limit to college reading skills or a threshold on the CAAP 
measurement of these skills. Thus, COD students may pick up a bit, but not fully, relative to 4- 
year public college students. 

The first-year to sophomore changes in reading scores are in marked contrast to the findings 
reported in the first year, preliminary assessments. Given the large number of cases and wider 
sampling, the current findings are considered representative. Two factors may play a role in 
what we can now classify as a substantial change. 

First, the developmental/remedial curriculum was changed just prior to the first round of testing 
in the fall of 1998. Based on the curriculum change and mandatory placement for students with 
low reading pre-tests, both the number of students enrolling in developmental reading classes, 
and the number of sections of these classes doubled in 1998-1999. These numbers have 
continued to increase in the last two years. Thus, at the onset of testing the percent of students 
with one or more developmental courses was lower than it is now after three years of higher 
participation rates. 

It is also possible, although there is no way to test the hypothesis, that after the targeting of 
college level reading skills in the fall of 1999, that changes in faculty pedagogies may have 
impacted student skill development. Responses after the announcement, and continuing 
anecdotal reports indicate a number of faculty reporting that they have made some changes in 
teaching strategies in order to help students strengthen their college reading skills. 
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• Table 30: Reading skills comparison 



College of DuPage Average 


Entering 

First-year 


Mid-studies 


Completing 

Sophomores 


N 


81 


147 


83 


Mean 


60.04 


60.06 


61.88 


Std. Deviation 


5.3 


5.3 


5.3 


Std. Error 


0.59 


0.44 


0.59 


95% Confidence Interval 


1.98 


1.98 


1.99 


Lower Bound 


58.86 


59.19 


60.71 


Upper Bound 


61.22 


60.93 


63.05 



2-Year Public College Average 



N 




1792 


21618 


Mean 




59.6 


61 


Std. Deviation 




5.1 


5.3 


Std. Error 




0.12 


0.04 


95% Confidence Interval 


1.96 


1.96 




Lower Bound 


59.36 


60.93 




Upper Bound 


59.84 


61.07 



4- Year Public College Average 



N 


4385 


30570 


Mean 


62.1 


63.0 


Std. Deviation 


5.2 


5.1 


Std. Error 


0.08 


0.03 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


61.95 


62.94 


Upper Bound 


62.25 


63.06 
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Mathematics skills 

Data from Table 31 indicates that COD first-year students enter with an average math skill similar 
to 2-year public freshman cohort. COD first-year students are significantly lower than the 4-year 
public freshman cohort. Math skills improve across their COD studies with significant gains 
between both first-year and mid-studies as well as between mid-studies and completion. COD 
completing sophomores' averages significantly exoeed the 2-year public college norms. That COD 
average also significantly exceeds the 4-year public college sophomore mean. That this gain occurs is 
laudable, at what cost may be a question as evident from articles in the student newspaper and 
student in-dass success rates. The conundrum of these multiple indicators indicates the presence of 
an issue broader than the inquiry of this review. 



• Table 31: Mathematics skills 



College of DuPage Average 


Entering 

First-year 


Mid-studies 


Completing 

Sophomores 


N 


81 


156 


78 


Mean 


56.62 


57.52 


59.24 


Std. Deviation 


4.23 


4.30 


5.12 


Std. Error 


0.46 


0.34 


0.58 


95% Confidence Interval 


1.98 


1.98 


1.99 


Lower Bound 


55.71 


56.84 


58.09 


Upper Bound 


57.53 


58.20 


60.40 



2-Year Public College 



N 


2828 


21221 


Mean 


56.3 


56.3 


Std. Deviation 


3.9 


3.5 


Std. Error 


0.07 


0.02 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


56.16 


56.25 


Upper Bound 


56.44 


56.35 



4- Year Public College 



N 


4554 


34639 


Mean 


58.1 


58.4 


Std. Deviation 


3.9 


4 


Std. Error 


0.06 


0.02 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


57.99 


58.36 


Upper Bound 


58.21 


58.44 
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Writing skills 

Data from Table 32: Writing skills(a multiple choice test) indicates that entering first-year COD 
students are similar to 2-year public college freshmen in their writing skills. These cases are 
significantly lower than 4-year public college freshmen. During their studies COD students 
improve their writing skills significantly by the time they are completing sophomores. Completing 
sophomores are similar to 2-year public college sophomores, but still significantly lower in their 
skills than their 4-year public college sophomore colleagues. 



• Table 32: Writing skills 



College of DuPage 
Average 


Entering 
First- year 


Mid-studies 


Completing 

Sophomores 


N 


81 


137 


93 


Mean 


60.65 


61.15 


62.80 


Std. Deviation 


5.19 


5.40 


5.42 


Std. Error 


0.58 


0.46 


0.56 


95% Confidence Interval 1.98 


1.98 


1.99 


Lower Bound 


59.50 


60.24 


61.68 


Upper Bound 


61.80 


62.07 


63.91 



2-Year Public College 
Average 



N 


2915 


23861 


Mean 


61.2 


62.6 


Std. Deviation 


5 


4.7 


Std. Error 


0.09 


0.03 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


61.02 


62.54 


Upper Bound 


61.38 


62.66 



4-Year Public College 



N 


5171 


35568 


Mean 


62.5 


64.5 


Std. Deviation 


5.6 


4.6 


Std. Error 


0.08 


0.02 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


62.35 


64.45 


Upper Bound 


62.65 


64.55 



O 
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Critical thinking skills 

Data from Table 33: Critical thinking skills indicates that the skills of COD students, while similar 
as both entering first-year students and completing sophomores to their 2-year public college 
colleagues, does not change significantly over their studies. It should be noted that while the 
averages improve, that improvement is not statistically significant. This lack of significant change 
may, however, be the result of small sampling size, constrains imposed by the statistical 
prooedure, or limits in sensitivity to change of the critical thinking area-test. The first-year to 
sophomore change in scores is examined in the next chapter using ANOVA, a more sensitive 
statistical test. The results reported on page 40 indicate that theses change do attain statistical 
significance. 



• Table 33: Critical thinking skills 



College of DuPage 
Average 


Entering 

First-year 


Mid-studies 


Completing 

Sophomores 


N 


94 


140 


87 


Mean 


59.27 


61.59 


61.48 


Std. Deviation 


5.06 


5.66 


5.68 


Std. Error 


0.52 


0.4780 


0.61 


95% Confidence Interval 1.98 


1.98 


1.99 


Lower Bound 


58.23 


60.65 


60.27 


Upper Bound 60.30 


62.54 


62.69 



2-Year Public College 
Average 



N 


4526 


18417 


Mean 


59.7 


61.1 


Std. Deviation 


5 


5.2 


Std. Error 


0.0743 


0.0383 


95% Confidence Interval 1.96 


1.96 


Lower Bound 


59.55 


61.02 


Upper Bound 59.85 


61.18 



4- Year Public College 
Average 



N 


5292 


8435 


Mean 


61.3 


62.3 


Std. Deviation 


5.2 


5.4 


Std. Error 


0.0715 


0.0588 


95% Confidence Interval 1.96 


1.96 


Lower Bound 


61.16 


62.18 


Upper Bound 


61.44 


62.42 
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Science reasoning skills 

Data from Table 34 indicates that COD's entering first-year students and completing sophomores 
are similar to their 2-year colleagues, and significantly lower than 4-year public college 
colleagues. Our students improve their science reasoning skills during their tenure at COD. It 
should be noted that although the test is labeled as science reasoning, examination of the test 
items indicate a test of concrete and logical reasoning and interpretation of data provided in 
charts, graphs, etc. These skills then are similar to not only life and natural science, but also 
many of the social and behavioral scienoes. 



• Table 34: Science reasoning skills 



College of DuPage Average 


Entering 

First-year 


Mid-studies 


Completing 

Sophomores 


N 


78 


134 


76 


Mean 


56.97 


58.49 


59.79 


Std. Deviation 


4.10 


4.65 


4.71 


Std. Error 


0.46 


0.40 


0.54 


95% Confidence Interval 


1.98 


1.98 


1.99 


Lower Bound 


56.05 


57.70 


58.71 


Upper Bound 


57.89 


59.29 


60.87 



2-Year Public College 



N 


1382 


13248 


Mean 


57.7 


59.0 


Std. Deviation 


4.1 


4.2 


Std. Error 


0.1103 


0.0365 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


57.48 


58.93 


Upper Bound 


57.92 


59.07 



4-Year Public College 



N 


3173 


25659 


Mean 


60.2 


61.1 


Std. Deviation 


4.5 


4.5 


Std. Error 


0.0799 


0.0281 


95% Confidence Interval 


1.96 


1.96 


Lower Bound 


60.04 


61.04 


Upper Bound 


60.36 


61.16 
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Essay writing skills 

Data from Table 35 indicates that COD's students are functioning at levels similar to 2-year 
freshmen and sophomores, and 4-year freshmen; but below the 4-year public college level. 
These data come from the first two of the three cydes of testing. After confirming that COD 
students were functioning at level, the Essay Writing testing was suspended since the instrument 
did not provide suffirient sensitivity to change in student essay skills. At this time the Student 
Outcomes Assessment Committee is discussing development of an in-dass essay sample 
procedure to replace the Essay Writing test from ACT. 



• Table 35: Essay writing 



College of DuPage Average 


Entering First- 
year 


Mid-studies 


Completing 

Sophomores 


N 


71 


135 


49 


Mean 


3.28 


3.23 


3.31 


Std. Deviation 


0.64 


0.66 


0.69 


Std. Error 


0.0765 


0.0565 


0.0980 


95% Confidence Interval 


1.98 


1.98 


2.01 


Lower Bound 


3.13 


3.12 


3.11 


Upper Bound 


3.34 


3.34 


3.51 


2- Year Public College 


N 


3223 




6535 


Mean 


3.2 




3.1 


Std. Deviation 


0.6 




0.6 


Std. Error 


0.0106 




0.0074 


95% Confidence Interval 


1.96 




1.96 


Lower Bound 


3.18 




3.09 


Upper Bound 


3.22 




3.11 


4- Year Public College 


N 


3389 




15982 


Mean 


3.0 




3.3 


Std. Deviation 


0.5 




0.6 


Std. Error 


0.0086 




0.0047 


95% Confidence Interval 


1.96 




1.96 


Lower Bound 


2.98 




3.29 


Upper Bound 


3.02 




3.31 
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First-year, mid-studies, completing sophomores - value added 



A second approach to considering the acquisition of general education skills focuses on changes 
from first-year students' skills to completing sophomores' skills. This approach to analysis utilizes 
dassification of cases based on quarter-hour credits at the time of testing. These dassifications 
were previously defined (see page 30). 

Two statistical tests can be considered as addressing this focus. The results of applying a t-test 
to examine significant change between entering-persisting first-year students and completing 
sophomores are listed in Table 36. The means for persisting first-year students and completing 
sophomores being compared are listed in Table 37. The changes from first-year to completion 
are significant for five of the six area-tests, with the Essay writing as the only area-test in which 
COD students do not show significant improvement. As mentioned previously, use of this test 
has been suspended due to a low level of sensitivity to change (see page 38). 

• Table 36: t-tests comparing first-year and completing sophomores 



Levene’s Test for 

Equality of Variances t-test for Equality of Means 







F 


Sig. 


t 


dfSig. (2-tailed) 


Writing Skills 


Equal variances 


.518 


.473 


-2.651 


172 


.009 


Score 


assumed 














Equal variances not 






-2.659 


170.404 


.009 




assumed 












Math Skills 


Equal variances 


5.432 


.021 


-3.570 


157 


.000 




assumed 














Equal variances not 






-3.556 


147.594 


.001 




assumed 












Reading Skills 


Equal variances 


.045 


.832 


-2.210 


162 


.029 




assumed 














Equal variances not 






-2.210 


161.917 


.029 




assumed 












Critical Thinking 


Equal variances 


2.363 


.126 


-2.776 


179 


.006 


Skills 


assumed 














Equal variances not 






-2.763 


172.570 


.006 




assumed 












Science 


Equal variances 


2.572 


.111 


-3.958 


152 


.000 


Reasoning Skills 


assumed 














Equal variances not 






-3.950 


147.986 


.000 




assumed 












Essay Skills 


Equal variances 


.063 


.802 


-.269 ~ 


118 


.788 




assumed 














Equal variances not 






-.266 


99.105 


.791 



assumed 



Comparisons among the three educational levels may also be of interest. Table 37 lists each of 
the six subject-area test score averages for the three educational levels of students. 

Comparisons of the averages for the three educational levels indicate that each of the six-subject 
area averages tended to increase. Five of the six changes were identified as statistically 
significant using ANOVA analysis. Changes in essay writing were not significant. Since this 
approach uses a more precise approach than the confidence intervals applied in the previous 
section, these findings may be more helpful to knowing when changes occur during a student's 
study. 
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In the case of writing skills there is significant change between first-year students and completing 
sophomores. There is a significant change between mid-studies and completing sophomore 
students, but not during the first year. Thus, it appears that these skills are developed 
throughout a student's tenure with acceleration during the second year. 

In the case of math skills there are significant changes between first-year students and 
completing sophomores. There is significant change between mid-studies and completing 
sophomores, but not first-year and mid-studies. Thus, it appears that these skills are developed 
throughout a student's tenure with acceleration during the second year. 

Significant changes in reading skills occur between entry and completion. It is interesting to note 
that significant changes only appear in the mid-studies to completing sophomore cohorts. The 
patterns of reading scores have made the most change over the past three years' reports. It is 
possible that program changes as discussed on page 32 may offer explanation for the instability 
of conclusions. At this point, after three years, the impacts of changed curriculum and faculty 
pedagogy may have worked their way through a major portion of the student body. Given these 
changes it should be noted that the effects from mandatory developmental reading courses and 
responses to the challenges of low reading skills identified in the first year's report have not 
stabilized. This is an area that will require continued evaluation in order to adequately identify 
both changes and conditions. 

Comparison of the conclusions based on t-tests and ANOVA analysis of Critical Thinking skills with 
those of the confidence intervals comparisons (on page 36), illustrate the increased sensitivity of 
the t-tests and ANOVA tests. Critical thinking skills increase significantly between first-year and 
completing sophomores. The increase from first-year to mid-studies is significant, while that 
from mid-studies to completing sophomore is not significant. 

A somewhat similar pattern emerges when examining Science Reasoning skill development. The 
change from first-year to completing sophomore is significant. The change from mid-studies to 
completing sophomore is not significant. The change from first-year to mid-studies is significant 
only if one applies a 10% willingness to accept a type I error. Given that both groups have over 
100 cases each, I find that .1 standard too generous, and conclude that the change from first- 
year to mid-studies is also not significant. 
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• Table 37: Summary of significant differences among COD education levels 



ANOVA F ratio and 
Scheffe Significance of Change 





N 


Mean 


First-year to 
mid-studies 


Mid-studies to 
completing 
sophomore 


First-year to 
completing 
sophomore 


Writing Skills 






F= 


4.021, sig. at 0 


H9 


First-year 


81 


60.65 








Mid-studies 


137 


61.15 


n.s. .802 






Comp. Soph. 


93 


62.80 




.075 


.033 


Total 


311 


61.51 








Math Skills 






F = 


7.141, sig. at .1 


301 


First-year 


81 


56.62 








Mid-studies 


156 


57.52 


n.s. .340 






Comp. Soph. 


78 


59.24 




.022 


.001 


Total 


315 


57.71 








Reading Skills 






F = 


3.576, sig. at .( 


329 


First-year 


81 


60.04 








Mid-studies 


147 


60.06 


n.s. .999 






Comp. Soph. 


83 


61.88 




.047 


n.s. .088 


Total 


311 


60.54 








Critical Thinking Skills 






F = 


5.904, sig. at .( 


303 


First-year 


93 


59.23 








Mid-studies 


140 


61.59 


.006 






Comp. Soph. 


87 


61.48 




n.s. .989 


.024 


Total 


320 


60.88 








Science Reasoning Skills 






F = 


7.480, sig. at .( 


301 


First-year 


78 


56.97 








Mid-studies 


134 


58.49 


n.s. .064 






Comp. Soph. 


76 


59.79 




n.s. .138 


.001 


Total 


288 


58.42 








Essay 






F = . 


,335, not sig. at 


.715 ... 


First-year 


71 


3.2782 








Mid-studies 


135 


3.2278 








Comp. Soph. 


49 


3.3112 








Total 


255 


3.2578 









As a general conclusion, we may accept that COD students make progress in gaining general 
education skills. These patterns of change support the generalization that general education 
skills development is a process that spans a number of oourses and years and is not the result of 
a limited number of specifically designated courses. The acquisition of these skills is in line with 
progress made at other 2-year public colleges as represented in the ACT'S published averages. 
However, it may be worth discussing whether being "average" is a sufficient goal for College of 
DuPage. 
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Table 38 lists the mean average for first-year persisting first-year students compared with those 
who stop-out. The averages are statistically significant in their differences for only age and math 
score. The Essay score differences could be argued as different if one aocepts a generous level 
of error. Among first-year students those who stop-out are older by 1.5 years, and they have 
substantially lower math skills. As one might predict for persisting students, class study time is 
higher and employment hours lower. Although not significant, it is interesting to note that both 
critical thinking and science reasoning are higher for stop-out than for persisting students. This 
may be a reflection of the difference in age, which supports the argument that these skills are 
developmental^ related to maturity and may not be total dependent on academic learning. 



• Table 38: First-year Comparison Persister v. Stop-out 





Native First- 
year outcomes 


N 


Mean 


St. 

Deviation 


Significance 


Age at testing 


Stop-out 


184 


9.77 


23.02 


T= 2.116 




Persister 


486 


7.59 


21.51 


Sig. .035 


Writing Skills Score 


Stop-out 


36 


5.46 


60.56 






Persister 


81 


5.19 


60.65 




Math Score 


Stop-out 


32 


4.78 


54.06 


T— 2.66 




Persister 


81 


4.12 


56.62 


Sig. .010 


Reading Score 


Stop-out 


21 


5.00 


58.71 






Persister 


83 


5.34 


60.00 




Critical Thinking Score 


Stop-out 


27 


5.86 


59.59 






Persister 


96 


5.07 


59.29 




Science Reasoning Score 


Stop-out 


33 


3.57 


58.00 






Persister 


78 


4.10 


56.97 




Essay 


Stop-out 


39 


0.87 


3.05 


T=-1.872 




Persister 


78 


0.69 


3.35 


Sig. .066 


Average class study time 


Stop-out 


146 


2.52 


3.48 






Persister 


428 


2.71 


3.86 




Employment (volunteer) 


Stop-out 


148 


1.24 


1.94 






Persister 


428 


1.17 


1.78 




Household work 


Stop-out 


146 


0.99 


1.19 






Persister 


426 


0.92 


1.15 
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Chapter 



Modeling the acquisition of general 
education skills 




The previous chapters of this analysis focused on the levels of general education skills 
development attained by specific classifications of College of DuPage students. This chapter will 
examine the relationship between measured student characteristics, actions, and courses with 
general education skills development. This analysis is organized in two ways. First, each of the 
six subject area models will be presented. Second, patterns of common influences and non- 
significant effects in general education skill acquisition will be identified starting on page 51. 



Modeling Method 



The models developed in this chapter are based on structural equations predictions. Model 
development begins with a fully spedfied model of twenty exogenous variables, nine of which are 
counts of courses in discipline areas, three of which are pre-test scores, three of which are hours 
spent on study, employment, and household work. The remaining five exogenous variables are 
gender, English as a second language, age, self-reported motivation on the test and credits 
earned at other colleges. Three intervening variables included are GPA earned at completion of 
the quarter when tested, total completed developmental credits and the total number of college 
credits (equal to or greater than 100) earned by completion of the quarter when tested. Rgure 1 
presents this model. Needless to say, this is a very complex model. 

The process of model building began with this full model with each of the five general education 
skills as the central dependent variable. First the full model was calculated using all available 
cases. While each case included values for most of the variables, some cases were missing 
values for some variables. In this initial modeling, parameters were estimated based on all 
available cases. Missing variable values were estimated using established maximum likelihood 
procedures. This estimation of missing values made the fullest use of all of the cases collected. 
Each of these full models utilized between five and six hundred cases. 

Each full model was then calculated a second time, using only the cases with complete data. 

Each of these models utilized between 153 and 179 cases. These complete data models were 
reduced to contain only significant paths and where possible to attain acceptable standards on 
two measures of fit. 

Models from these two approaches were compared for substantial differences and patterns of 
similarity. These observations were then applied to develop a final model for each of the five 
general education skills areas that use all of the cases. In one model one non-significant path 
was retained in order to test or illustrate a "common sense" connection of specific interest. 
Finally, parameters on some of the exogenous variables were relaxed so that the models fit the 
data. 

While the process of calculating and building these models is rampant with mathematical 
processes, standards and parameters forjudging the models are still emerging from active 
exploration. The models presented need to be considered as best fit descriptions of the 
processes but far from perfect measures of the underlying processes. Their development is a 
mix of statistical modeling and theory with intuition. These models are best used for discussion, 
dialogue, and generating next inquiry hypothesis. The causal assumptions implied in the 
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statistical models are imposed by this researcher, and not tested through the data. Thus, theory 
and experience enter the specification of these models. 




• Figure 1: Full Model 

Two patterns of relationships occur in the models. In one of these patterns specific course work 
contributes directly to skill development. This pattern is most evident in the model for acquisition 
of mathematics skills. This is the first model presented. The acquisition of science reasoning 
skills model also illustrates this direct course pattern, but the course categories with direct impact 
are not just science oriented. 

In the second of these patterns the general accumulation of course credits (at or greater than 
100) contributes to skill development, while specific course impacts are either smaller or 
negative. This pattern is most evident in the model for acquisition of critical thinking skill, the 
third model presented. It is also evident in the models for acquisition of reading skills and writing 
skills. 

The description of the model for acquisition of math skills is presented with the most detail about 
how to interpret and examine these models. Even if one is not interested in the math model, 
reviewing its presentation is the most instructive for interpreting all of the other models that 
follow. 
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Acquiring mathematics skills 



The process of modeling is dearly illustrated by the model examining math skills. Figure 2 
illustrates the reduced Mathematics Model. Straight, single headed arrows indicate statistically 
significant paths connecting variables. The path weight is a standardized regression coefficient, 
which may be interpreted as the standardized effect of the originating variable on the target 
variable controlling for the other variables in the model as indicated by arrows. These paths 
range from -1 to 1, although those paths nearest 0 are not significant and have been removed 
from the model. The use of standardized coeffidents means that path coefficients can be 
compared with one another. In general, standarized coeffidents can be interpreted similar to 
correctiion coefffidents with one added idea. The coeffidents in these models indicate the 
relationship controlling for the other significant (and specified) variables in the model. 

In the math model, the path from courses to math skills is .22. This coeffident is judged to be 
moderately strong. This coeffident can be interpreted as indicating that controlling for math pre- 
test scores, the more courses in mathematics the higher the level of math skills as measured by 
the CAAP test. "Well, of course," one might say. But, as discussed later, this pattern of direct 
impact from specific course counts is not found in many of the other skills areas. 

The courses-in-math to math-skills path is stronger than the path from science courses to math 
skills that is .11. The path from the Z score of the math pre-tests at .47 is the strongest path in 
the model, and thus the best single predictor of math skill performance. 

The curved arrows with heads at either end indicate covariance between variables, which are 
accepted in calculating the model. Accepting this covariance between exogenous variables is 
necessary to estimating the model. The covariance between endogenous variables can be 
indicative of unspedfied relationships or problems with models or it may be a necessity for 
estimating the model. The accepted covariance paths are presented in the model for the 
purpose of foil reporting; however, they are judged as acceptable necessities not interesting 
influences and will not be discussed further. 

The model fits the data reasonable well and the variation in the independent and intervening 
variables in the model (z-pretest algebra, z-pretest reading, courses in saence and math, GPA, 
study-time, employment, household work, gender, age, motivation, and other college credits 
earned) account for 54% of the variation in the math skills scores. This is the highest level of 
explained variance among the five models. This explained variance (squared multiple 
correlation) is the number presented in the model at the upper right comer of intervening 
variables and the dependent variable. 
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Math Skills 

ChiSq= 267.985, df= 57, prob. = .000, cfi= .980 




• Figure 2: Mathematics Model 
When examining the mathematics model three issues emerge. 

o The model for mathematics acquisition indicates that courses in mathematics and science 
tend to increase math skills. There is evidence that courses enhance skills controlling for 
entering skill levels as measured by the pre-test. 

o There is also evidence that these learning effects are tempered by a negative impact of 
both age (-.23) and gender (-. 12 for females). The negative impact of age is one of the 
stronger paths in the model. 

o This is the only one of the five models in which employment has both an indirect effect 
(-.09) and a direct effect (-.07) on skills acquisition. This may reflect the on-the-street 
word that "You have to do the homework if you want to leam the skills." That frequent 
homework takes time, something that competes with employment. 



Acquiring science reasoning skills 



Figure 3 illustrates the reduced Science Reasoning model. The model fits the data reasonably 
well and the variations in the independent variables and intervening variables in the model 
account for 46% of the variation in the science reasoning skills scores. The path from courses in 
science to science reasoning is .10 and that from math courses to science reasoning is .08. It is 
interesting to note that the number of vocational courses also has a positive effect (.08) on 
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science reasoning. This may be due in part to the nature of the test skills, which involve 
concrete and physical world problems. 



Science Reasoning Skills 

ChiSq= 91.016, df= 43, prob. = .000, cfi= .994 




• Figure 3: Science Reasoning Model 

When examining the science reasoning model one conclusion can be supported. 

o The model for science reasoning acquisition indicate that course in science, math, and 
technical/vocational areas tend to increase science reasoning skills. 



Acquiring critical thinking skills 



With this model we examine a second pattern in general education skills acquisition, that of 
accumulated courses across the curriculum. Figure 4 illustrates the reduced Critical Thinking 
model. The model fits the data reasonably well and the variations in the independent variables 
and intervening variables in the model account for 36% of the variation in the critical thinking 
variable. This is the smallest explained variance among the five models being examined. The 
path from credit hours at or above 100 to the critical thinking skills score is .17, the highest 
coefficient for this relationship among all of the models. The only specific course impact in the 
model is a path from business courses to critical thinking; however, that path has a negative 
coefficient of -.08. This implies that those students with more business classes tend to have 
lower critical thinking scores controlling for the effects of pre-test scores, GPA, and being a native 
speaker of English. 
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The path from GPA to critical thinking skills is .19, the highest coefficient for this path among all 
of the models. This implies that a portion of the performance measured by GPA is a 
measurement of critical thinking skills applied within the oourse. 



Critical Thinking Skills 

ChiSq= 88.748, df= 35, prob. = .000, cfi= .993 




• Rgure 4: Critical Thinking Skill Model 



When examining the critical thinking model three issues emerge. 

o The model for critical thinking skills acquisition indicates that the accumulation of courses 
in almost all categories tends to improve critical thinking skills. 

o One possible explanation for this building of skills might be increasing maturity. 

However, this explanation is not supported since there is no significant relationship with 
age controlling for the other variables in the model. 

o Time spent studying has no direct effect on critical thinking skills; however, study time 
does have a strong impact on GPA (.21 which means higher study time, higher grades). 



Acquiring of reading skills 



Rgure 5 illustrates the reduced Reading Skills model. The model fits the data reasonably well 
and the variations in the independent variables and intervening variables in the model account 
for 38% of the variation in the reading skills scores. The model in Rgure 5 contains two paths 
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from course counts, one from communication and one from humanities. The path from 
communication to reading skills is -.08 and the ratio of the coefficient to its standard error is 
-1.773 indicating a non-significant path. The paths from humanities to reading skills and from 
credits at or above 100 to readings skills are significant. If the model is further reduced by 
removing the communications reading skills path, then the humanities path become non- 
significant as does the credit >= 100. Removing the communication and humanities paths 
returns the model to approximately the same level of explanation as that in Figure 5 with the 
courses >=100 significant. Here then one can see that the issues of model building are not just 
mathematical in nature. Based on seeking an explanative model that is true to the patterns in 
the data. I'm reporting the model including both counts of communication (not significant) and 
humanities (significant). 

Reading Skills 

ChiSq= 120.581, df= 50, prob. = .000, cfi= .992 




• Figure 5: Reading Skills Model 



When examining the reading skills model four issues emerge. 

o The model for reading skills acquisition indicates that accumulation of course work tends 
to increase college-reading skills. 

o In addition to the effect of accumulation of courses, courses in humanities increase and 
courses in communication decrease, thus off-setting each other, reading skills 
acquisition. 

o Time spent studying tends to increase GPA. And controlling for GPA, students spend less 
time studying when their reading skills are higher. 
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o Students spending more time employeed or volunteering have higher reading skills. 



Acquiring writing skills 



Figure 6 illustrates the reduced Writing Skills model. The model fits the data reasonably well and 
the variations in the independent variables and intervening variables in the model acoount for 
41% of the variation in the writing skills scores. The path from accumulated courses >=100 to 
writing skills is .10, indicating a moderate impact on skills by courses completed. The path from 
communication courses to writing skills of -.10 is less interpretable. Some background may 
provide one possible explanation for this observation. 

This area-test of the CAAP battery is a multiple choice format test. Some critics describe the 
writing skills tested in this area-test have been described as "editing" skills. Our prior use of the 
Essay test area-test, a test utilizing a writing sample, showed it to be insensitive to changes in 
student skills. Given that most students take only a three course sequence of English 
composition along with speech, it may be that those taking more courses in the communication 
area are those inclined to reject traditional rhetorical rules. Thus, the few students with more 
communication courses may also be the students with more non-traditional styles and thus less 
likely to score well on a traditional English rhetoric exam. An alternative explanation for this 
counter intuitive observation may lie in some complexity of the basic model being imposed. 
Exploration of this issue is not undertaken at this time. 



Writing Skills 

ChiSq= 172.647, df= 45, prob. = .000, cfi= .987 




• Figure 6: Writing Skills Model 
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When examining the writing skills model two issues emerge. 

o The model for writing skills acquisition indicates that the general accumulation of courses 
positively impacts writing skills. 

o The impact of GPA on writing skills is equal to that in the critical thinking model (.19) 
indicating that writing skills performance is an important component of course 
performance. 



Pretest and entry level skills 



One of the issues central to assessment is the challenge of identifying effects attributable to a 
setting or event in contrast to those effects that are external to the setting or event. In response 
to this challenge these models use as control variables the z-scores of the pre-tests in English, 
Algebra, and Reading. It should be noted that the variable with the highest rates of missing data 
are missing pre-test soores. In future years inclusion of results from the Prarie State testing 
efforts and increased collecting of pre-tests from all students should help strenthen control for 
entry level skills. 

The standardized coefficients from the significant paths between these pre-tests and each of the 
five dependent variables from the five models are listed in Table 39. The relative size of these 
coefficients indicates a substantial impact of prior skills on current skills. This supports a 
commitment to controlling for entry-level skills as an important part of modeling are skills 
acquisition or value added assessment. 






Ta ble 39: Significant Standardized Coefficients - Pretests 





Math 


Saence 


GThink 


Read 


Write 


Z-English 




0.10 




0.12 


0.18 


Z-Algebra 


0.47 


0.38 


0.22 


0.19 


0.17 


Z-Reading 




0.22 


0.30 


0.19 


0.13 



Time commitments and outcomes 



One of the issues frequently discussed with reference to community college students' educational 
achievement is the issue of competing time commitments and role-strains resulting form high 
levels of employment and family relationships that differ from traditional students. To examine 
these issues we inducted three questions focused on the time commitments to 1) studying for 
dasses, 2) work, and 3) family commitments. These three variables were collected for students 
taking five of the six area tests. Respondents taking the Essay test did not consistently answer 
the institutional designated questions. 

Table 40 lists the significant standardized coeffidents of time commitments from each of the five 
models. In all five models the impact of study time on GPA is significant with a range from .13 to 
.21. All of these are moderate impacts. Also in all five of the models the connection of GPA with 
the general education skills is significant and range from .09 to .19. The coeffidents for critical 
thinking, writing, and reading skills are similar and moderate. The remaining coeffidents appear 
more unique to the speafic models and tend to be weak in effect. 
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• Table 40: Significant Standardized Coefficients - Time 





Study 


Employment 


Household 




Study 


Employment 


Household GPA 


GPA Math 


0.16 


-0.09 


.10 


Math 




-0.07 


0.15 


GPASR 


0.13 




0.17 


SR 






0.09 


GPA CT 


0.21 




0.12 


cr 






0.19 


GPA Read 


0.17 






Read 


-0.12 


0.08 


0.18 


GPA Write 


0.20 






Write 




0.09 


0.19 



What then might be the important issues to note? 

o Study time does have a direct impact of grade performance. Studying is recognized and 
rewarded with higher grades. 

o General education skills are integrated and inherent parts of oourse work and grades in 
part reflect the levels of these skills students are able to bring to courses. 

o Unlike our general common sense intuition, employment and household work do not 
appear to have a direct importance on grade performance, subject learning and 
acquisition of general education skills. Here we need to be careful to recognize that 
although specific individual students may find time and role conflicts a significant factor 
in their personal success in course work, there are other students with the same time 
conflicts and demands who acquire skills despite those challenges. 



Motivation and demographic variables 



For the first time this year our models include consideration of the self-reported motivation of the 
students taking these tests. The standardized path coefficients between motivation and the 
general education skills scores range from a high of .33 in the writing model to .15 in the math 
model. The coefficients in three of the models (critical thinking, science reasoning, and reading) 
are either .23 or .22. In all of the models these paths are among the strongest effects reported. 
Thus, the positive effect of motivation on student performance on the CAAP test is confirmed. 

An interesting contrast is evident with the low coefficient in the math model. It appears that in 
the case of testing math skills, students either are able to complete or fail the testing challenges 
based on knowledge, and their scores are less impacted by motivation than in the other area- 
tests. 

Given concerns among assessment practitioners about standardized testing and students' efforts 
and motivation, it may be important to consider how we should judge this factor. The implication 
for testing efforts appears to be a challenge to motivate student effort, while accepting that a 
range of efforts is demonstrated among all test takers. To the extent motivation is higher in a 
specific setting, that set of scores can be expected to out-shine results from similar students with 
lower motivation. 

It is thus apparent that each report of student outcomes performance must be placed within the 
context of the testing setting and campus climate. Assessment researchers need to develop the 
controls appropriate to consider skills performance within the context of testing controlling for 
motivation. This report undertakes to report that context in our examination beginning on page 
12 . 
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A focus for assessment of general education skills development at College of DuPage begins with 
the definition of general education. Based on our catalogue statement seven outcomes emerge. 
While it is our intent to gather evidence focusing on each of these outcomes, no single 
assessment measure provides a complete examination. The first three outcomes have, thus far, 
been assessed only through self-reports from students. Given that these are self-reported, the 
rankings may be considered as reflective evaluations rather than direct behavioral indicators. 

The next three outcomes are more easily examined using objective measures since they are skills 
based. Using the patterns of evidence gathered through three years of general education skills 
testing using the ACT-CAAP standardized tests, some general conclusions can be drawn. The 
following items summarize conclusions based on three year's of outcomes assessment. The 
averages reported in these conclusions are those for freshmen and sophomores at 2-year public 
colleges and 4-year public colleges and universities. In this report they are referred to as 
"national average." The averages used are those most recently published as the Fall, 2000 CAAP 
User Norms. 

The aims of general education are to enable students to: 

1. Understand and appreciate their culture 

Among completing sophomores, 33.7% report their Gollege of DuPage courses have 
meaningful or significant impact on their cultural appreciation. This contrasts with 
18.5% to 20.6% among entering first-year and mid-studies students. Although only 
about one-third of the students report significant impact, the change indicates a 
significant increase in students' evaluation through accumulating courses. 

2. Understand and appreciate their environment 

Among completing sophomores, 30.5% report their courses have meaningful or 
significant impact on their environmental appreciation. This ranking is substantially 
higher than the 16.5% ratings among mid-studies students but lower than the 33.6% 
ranking among entering first-year students. While the percent ranking at the top two 
levels does not increase, the percent choosing the lowest two categories decreases with 
a substantial increase in the moderate impact ranking. 

3. Develop a system of personal values based on accepted ethics that lead to 
civic and social responsibility 

Among completing sophomores, 39.2% report their courses have meaningful or 
significant impact on their developing a system of personal values based on aooepted 
ethics that lead to dvic and soaal responsibility. This contrasts with rankings of 29.5% 
among entering first-year and 21.7% among mid-studies students. The U-shaped 
reports of impacts may reflect first flush of exposure to oollege-level thinking, followed by 
mid-studies uncertainty and eventually beginning development of more complex ethical 
positions. 

In each of these three assessments the restrictive nature of self-reports needs to be considered. 
These data may provide a beginning baseline for consideration and discussion. 

The general education goals 4 through 6 are skills areas where direct indications of student 
profidency were obtained from the ACT-CAAP testing over the past three years. 
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4. Attain skills in analysis 

Assessment of skills development in analysis may be inferred from two of the subject 
area tests in the CAAP battery: critical thinking and scientific reasoning. Results from the 
Critical Thinking test indicate that our first-year students are similar to other 2-year 
students both of who are lower in skills than 4-year students. On average our 
completing sophomores demonstrate significantly higher critical thinking skills than our 
first-year students (t= -2.763, sig. .006). The COD sophomore average is similar to 2- 
year sophomores and is not statistically different from the 4-year sophomore average. 

Results from the Scientific Reasoning test indicate that our first-year students are similar 
to other 2-year students both of who are lower in skills than 4-year students. On 
average our completing sophomores have improved significantly (t= -3.95, sig. .000). 
The COD sophomore average is similar to 2-year sophomores but significantly lower than 
4-year sophomores. 

Thus, in the area of developing general education skills in analysis as indicated by these 
two area tests, the students attending COD are in-step with other 2-year public college 
students. 

5. Attain skills in communications 

The broad goal of developing skills in communication is frequently divided into four more 
specific skills that are described by two dimensions. One of these dimensions is the 
format dimension of written and oral; the second dimension is the modality of receiving 
or producing. The four skills are reading, writing, listening, and speaking. CAAP skill- 
area tests cover two of these. In the area of writing skills, CAAP has both a multiple- 
choice instrument and a writing sample essay instrument. In the area of reading, CAAP 
examines college level skills in both the humanities and social sciences. The current 
general education skill development assessments do not include measures focused on 
listening nor on speaking skills. 

Results from the college Reading indicate that our first-year students are similar to other 
2-year students both of who are lower in skills than 4-year students. On average our 
completing sophomores have significantly improved (t= -2.21, sig. .029) from first-year 
averages. The COD sophomore average is similar to both 2-year and 4-year sophomores 

Results from the Writing Skills tests indicate that our first-year students are similar to 
other 2-year students both of who are lower in skills than 4-year students. On average 
our completing sophomores have significantly improved (t= -2.659, sig. .009). The COD 
sophomore average is similar to 2-year sophomores but still significantly lower than 4- 
year sophomores. 

Results from the Essay test (a writing sample conducted from 1998 to 2000) indicate that 
our first-year students are similar to the 2-year college average both of who are higher in 
skills than the 4-year student average. On average our sophomores demonstrated no 
significant change (t= -.269, sig. .791). The COD completing sophomore average was 
significantly higher than the 2-year college average and is not statistically different from 
the 4-year sophomore average. Use of this test was suspended in the foil of 2000 after it 
was determined that we were at level and that the test lacked sensitivity to changes in 
students' skills. SOAC will be working to develop an alternative, effective writing sample 
instrument. 
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6. Attain skills in quantification - mathematics 

General education skills development in quantification is most directly linked to outcomes 
measured by the Mathematics CAAP area-test. Results indicate that our first-year 
students start out similar to other 2-year public college students both of whom are lower 
in skills than 4-year public college students. On average our completing sophomores 
have improved their math skills significantly (t= -3.57, sig. .000). The COD sophomore 
average is significantly higher than 2-year sophomores and similar to that of sophomores 
at 4-year public colleges. 

Student Outcomes Assessment has yet to develop an indicator measure for the seventh general 
education focus. 

7. Attain skills in synthesis 

Assessment of the seventh general education outcome requires gestalt approaches that 
have not yet been designed nor implemented. Some indication of student, employer, 
and community satisfaction in this area is possible from other assessments including 
student satisfaction surveys and community needs studies. 
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Summary of conclusions 



_/L The strongest pattern of development of general education skills by College of 
DuPage students is acquisition of mathematics -- quantification skills. 

£-^=j Also evident is a pattern of improvement in writing skills as measured through 
multiple-choice items covering understanding of conventions of standard written 
English. 



Our students demonstrate significant increases in their science reasoning. 

Although our students perform at national two-year norms, minimal significant 
change occurs in college reading skills. Improving college-level reading skills is 
probably the single most important outcome. It's worthy of support because 
college-level reading skills are direct prerequisites to critical thinking and 
reasoning. 



First-year students enter and exit at two-year national average in their critical 
thinking skills. There is significant improvement in their critical thinking skills. 
This general education skill is one that can be developed in a wide variety of 
content and skills based courses. 



No significant change occurs in development of essay writing skills as measured 
in a writing sample (which is similar to the pattern in the national norms). 



The bottom line conclusion justified by these observations is that College of DuPage students are 
similar to other 2-year public community college students. However, it may be worth discussing 
whether being "average" is a sufficient goal for College of DuPage. Recognizing and 
implementing some dassroom changes in pedagogy can increase our students' general education 
skills. Such development should focus on ways in which faculty can improve subject-area 
learning through supporting general education skills development across the curriculum. 
Improving general education skills development is a key to increasing student understanding and 
knowledge of content area and for the mid-range of our student body that development is best 
accomplished within the context of their subject-area courses. 
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